Nov 01 00:20:49.665576 ip-10-0-141-93 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 00:20:49.665585 ip-10-0-141-93 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 00:20:49.665591 ip-10-0-141-93 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 00:20:49.665834 ip-10-0-141-93 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 00:20:59.914801 ip-10-0-141-93 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 00:20:59.914819 ip-10-0-141-93 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 642623d84185403a82e3ec92f3e40cc5 -- Nov 01 00:23:02.024672 ip-10-0-141-93 systemd[1]: Starting Kubernetes Kubelet... Nov 01 00:23:02.362437 ip-10-0-141-93 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:23:02.362437 ip-10-0-141-93 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 00:23:02.362437 ip-10-0-141-93 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:23:02.362437 ip-10-0-141-93 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:23:02.368449 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368361 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:02.368449 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368446 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:02.368449 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368450 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:02.368449 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368453 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368456 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368459 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368462 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368464 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368467 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368470 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368472 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368475 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368477 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368480 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368482 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368485 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368493 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368496 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368499 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368502 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368505 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368508 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368511 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:02.368570 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368513 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368516 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368518 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368520 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368523 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368526 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368528 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368531 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368533 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368536 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368538 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368542 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368544 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368548 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368550 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368553 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368556 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368559 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368561 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368564 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:02.369067 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368566 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368569 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368572 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368575 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368578 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368580 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368583 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368586 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368591 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368596 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368599 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368601 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368604 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368606 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368609 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368611 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368614 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368616 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368619 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:02.369556 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368621 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368623 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368626 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368629 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368632 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368635 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368638 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368641 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368644 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368648 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368653 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368657 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368660 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368664 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368667 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368669 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368672 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368675 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368678 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:02.370105 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368681 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368683 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368686 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368688 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368692 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368694 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368697 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368699 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368702 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368704 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368707 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368709 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368712 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368714 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368717 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368720 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368722 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368725 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368728 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368731 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:02.370579 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368734 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368736 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368739 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368741 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.368744 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369549 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369556 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369559 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369563 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369566 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369569 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369572 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369575 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369577 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369580 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369582 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369585 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369588 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369590 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369593 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:02.371049 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369596 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369598 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369601 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369604 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369608 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369610 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369613 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369616 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369619 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369622 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369624 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369627 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369630 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369633 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369635 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369638 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369640 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369644 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369646 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:02.371602 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369649 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369651 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369654 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369656 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369659 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369661 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369664 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369666 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369669 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369671 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369674 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369676 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369679 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369681 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369684 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369686 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369689 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369692 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369694 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:02.372059 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369697 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369700 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369702 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369705 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369707 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369711 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369714 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369717 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369719 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369721 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369724 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369726 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369729 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369732 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369735 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369737 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369740 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369743 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369745 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369747 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:02.372523 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369750 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369753 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369755 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369758 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369760 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369763 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369765 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369768 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369771 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369774 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369777 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369780 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369782 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369785 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369788 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369793 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369797 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369800 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369803 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:02.372994 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369806 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369808 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369811 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369813 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369815 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369818 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369821 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369823 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369826 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369828 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369831 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369833 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369836 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.369838 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370338 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370344 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370347 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370350 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370353 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370356 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:02.373476 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370358 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370361 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370364 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370366 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370369 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370372 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370374 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370377 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370379 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370382 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370385 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370387 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370390 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370393 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370395 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370398 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370400 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370403 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370406 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370409 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:02.373940 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370412 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370414 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370417 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370420 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370422 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370425 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370429 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370432 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370436 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370439 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370442 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370447 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370449 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370452 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370455 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370457 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370461 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370466 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370469 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:02.374424 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370471 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370474 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370477 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370479 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370482 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370484 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370487 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370490 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370492 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370495 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370498 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370500 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370510 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370513 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370516 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370518 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370521 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370523 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370526 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370529 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:02.374867 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370531 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370534 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370536 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370539 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370543 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370545 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370548 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370551 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370553 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370556 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370558 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370561 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370564 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370566 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370568 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370571 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370573 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370576 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370578 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:02.375352 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370581 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370584 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370586 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370589 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370592 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370594 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370602 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370605 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370608 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370610 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370613 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370615 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370618 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370620 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370623 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370625 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370628 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370632 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370634 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:02.375801 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370637 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370640 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.370643 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371304 2570 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371313 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371323 2570 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371328 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371333 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371336 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371340 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371345 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371348 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371351 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371354 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371358 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371361 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371364 2570 flags.go:64] FLAG: --cgroup-root="" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371366 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371370 2570 flags.go:64] FLAG: --client-ca-file="" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371372 2570 flags.go:64] FLAG: --cloud-provider="external" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371376 2570 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371383 2570 flags.go:64] FLAG: --cluster-domain="" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371386 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371389 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 00:23:02.376245 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371392 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371395 2570 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371399 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371402 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371405 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371408 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371413 2570 flags.go:64] FLAG: --contention-profiling="false" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371416 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371419 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371423 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371426 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371431 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371434 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371437 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371439 2570 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371442 2570 flags.go:64] FLAG: --enable-server="true" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371445 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371451 2570 flags.go:64] FLAG: --event-burst="100" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371455 2570 flags.go:64] FLAG: --event-qps="50" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371458 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371461 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371463 2570 flags.go:64] FLAG: --eviction-hard="" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371467 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371471 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371474 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 00:23:02.376849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371477 2570 flags.go:64] FLAG: --eviction-soft="" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371480 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371483 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371486 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371489 2570 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371492 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371495 2570 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371498 2570 flags.go:64] FLAG: --feature-gates="" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371502 2570 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371505 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371508 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371511 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371514 2570 flags.go:64] FLAG: --healthz-port="10248" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371518 2570 flags.go:64] FLAG: --help="false" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371521 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371525 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371528 2570 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371531 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371534 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371538 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371541 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371543 2570 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371546 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 00:23:02.377465 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371549 2570 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371552 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371555 2570 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371558 2570 flags.go:64] FLAG: --kube-reserved="" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371561 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371564 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371567 2570 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371569 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371572 2570 flags.go:64] FLAG: --lock-file="" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371575 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371578 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371581 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371586 2570 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371589 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371593 2570 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371596 2570 flags.go:64] FLAG: --logging-format="text" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371599 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371602 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371605 2570 flags.go:64] FLAG: --manifest-url="" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371608 2570 flags.go:64] FLAG: --manifest-url-header="" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371613 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371616 2570 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371620 2570 flags.go:64] FLAG: --max-pods="110" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371623 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371628 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 00:23:02.378054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371631 2570 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371634 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371637 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371640 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371643 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371651 2570 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371654 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371657 2570 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371660 2570 flags.go:64] FLAG: --pod-cidr="" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371663 2570 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371666 2570 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371669 2570 flags.go:64] FLAG: --pods-per-core="0" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371672 2570 flags.go:64] FLAG: --port="10250" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371675 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371677 2570 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-015c297573811e9b6" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371681 2570 flags.go:64] FLAG: --qos-reserved="" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371684 2570 flags.go:64] FLAG: --read-only-port="10255" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371686 2570 flags.go:64] FLAG: --register-node="true" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371689 2570 flags.go:64] FLAG: --register-with-taints="" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371696 2570 flags.go:64] FLAG: --registry-burst="10" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371698 2570 flags.go:64] FLAG: --registry-qps="5" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371701 2570 flags.go:64] FLAG: --reserved-cpus="" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371704 2570 flags.go:64] FLAG: --reserved-memory="" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371708 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371711 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371714 2570 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 00:23:02.378684 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371717 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371720 2570 flags.go:64] FLAG: --runonce="false" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371723 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371726 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371729 2570 flags.go:64] FLAG: --seccomp-default="false" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371733 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371736 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371739 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371742 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371745 2570 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371748 2570 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371751 2570 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371754 2570 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371757 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371760 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371763 2570 flags.go:64] FLAG: --system-cgroups="" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371766 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371772 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371781 2570 flags.go:64] FLAG: --tls-cert-file="" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371784 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371788 2570 flags.go:64] FLAG: --tls-min-version="" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371791 2570 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371793 2570 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371796 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371799 2570 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 00:23:02.379314 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371802 2570 flags.go:64] FLAG: --v="2" Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371807 2570 flags.go:64] FLAG: --version="false" Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371811 2570 flags.go:64] FLAG: --vmodule="" Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371815 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.371823 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371966 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371969 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371972 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371975 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371978 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371980 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371983 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371988 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371990 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371993 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371996 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.371998 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372001 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372003 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372006 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372008 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:02.379908 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372011 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372013 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372016 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372018 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372021 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372023 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372026 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372028 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372031 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372033 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372036 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372038 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372041 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372043 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372046 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372048 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372051 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372054 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372057 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372059 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:02.380730 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372062 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372065 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372068 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372072 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372074 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372077 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372080 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372082 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372085 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372087 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372090 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372093 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372095 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372098 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372100 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372102 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372105 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372107 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372110 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:02.381496 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372112 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372115 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372117 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372120 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372122 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372125 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372127 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372130 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372132 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372135 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372138 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372141 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372144 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372147 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372149 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372152 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372156 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372158 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372161 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372163 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:02.381950 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372166 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372168 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372172 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372174 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372177 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372179 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372183 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372187 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372190 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372193 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372196 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372198 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372201 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372204 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372206 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372209 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372211 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372214 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372216 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:02.382451 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372219 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372221 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372225 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372229 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372234 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372237 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372240 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372243 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372246 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372250 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372252 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372255 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.372294 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372330 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372335 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372338 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:02.382893 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372340 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372343 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372346 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372348 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372352 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372354 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372357 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372359 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372362 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372364 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372367 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372369 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372372 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372374 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372377 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372379 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372382 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372385 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372387 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372390 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:02.383260 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372392 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372397 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372399 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372402 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372405 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372408 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372411 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372413 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372416 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372419 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372422 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372424 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372427 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372429 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372432 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372434 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372437 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372439 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372442 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:02.383744 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372445 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372447 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372450 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372452 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372454 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372457 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372459 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372462 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372464 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372467 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372469 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372472 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372474 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372477 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372481 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372484 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372487 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372489 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372494 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:02.384193 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372497 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372499 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372502 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372504 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372507 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372510 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372512 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372515 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372517 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372520 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372522 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372525 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372528 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372531 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372533 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372537 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372541 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372544 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372547 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372549 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:02.384653 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372552 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372554 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372558 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372561 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372563 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372566 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372568 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372572 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372575 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372578 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372581 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372586 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372589 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372592 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372595 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372598 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372601 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372603 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372606 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:02.385122 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372609 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372611 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372614 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372616 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372619 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.372621 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.379341 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.379473 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.379488 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.379493 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379567 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379572 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379575 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379578 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379582 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379584 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379587 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379589 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379592 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:02.385597 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379596 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379598 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379601 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379604 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379606 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379609 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379611 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379614 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379617 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379619 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379622 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379625 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379627 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379630 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379632 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379635 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379638 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379641 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379643 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:02.386089 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379646 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379650 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379653 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379656 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379658 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379661 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379663 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379666 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379668 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379671 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379673 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379676 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379689 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379692 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379695 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379698 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379700 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379703 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379706 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:02.386561 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379708 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379711 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379713 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379716 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379718 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379723 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379727 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379731 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379734 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379737 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379739 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379742 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379745 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379748 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379751 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379754 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379757 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379760 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379762 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:02.386993 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379765 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379767 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379770 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379772 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379775 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379778 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379780 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379782 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379785 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379787 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379790 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379792 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379795 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379797 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379800 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379802 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379805 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379807 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379810 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379812 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:02.387492 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379815 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379817 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379820 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379823 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379827 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379829 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379833 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379835 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379838 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379841 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379844 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379846 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379849 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379851 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379854 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379857 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379859 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379862 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379864 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:02.387955 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.379867 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.379896 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380037 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380041 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380045 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380047 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380050 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380053 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380055 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380058 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380061 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380065 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380068 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380071 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380074 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380077 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:02.388427 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380079 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380082 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380085 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380087 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380090 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380093 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380095 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.380142 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381451 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381465 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381471 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381476 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381481 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381486 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381493 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381498 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381522 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381616 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381621 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381624 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:02.388798 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381626 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381629 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381632 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381635 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381637 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381640 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381643 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381646 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381649 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381651 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381654 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381657 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381659 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381662 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381665 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381667 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381670 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381672 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381675 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381677 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:02.389261 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381680 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381685 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381688 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381691 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381693 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381696 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381699 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381701 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381704 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381706 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381709 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381711 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381714 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381716 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381719 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381722 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381724 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381727 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381730 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381733 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:02.389914 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381736 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381738 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381741 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381743 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381746 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381748 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381750 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381753 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381755 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381760 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381764 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381767 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381770 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381773 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381776 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381778 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381781 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381784 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:02.390426 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381786 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381789 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381792 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381794 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381797 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381800 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381803 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381805 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381810 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381813 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381815 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381820 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381822 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381826 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.381866 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381908 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:02.390891 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381913 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381916 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381919 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381921 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381923 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381926 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381929 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381931 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381934 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381936 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381939 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381942 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381945 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381947 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381950 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381953 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381956 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381960 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381965 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:02.391347 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381970 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381973 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381975 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381978 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381981 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381983 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381985 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381988 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381991 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381993 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381996 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.381998 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382001 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382003 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382005 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382008 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382010 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382013 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382015 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382018 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:02.391803 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382022 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382024 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382027 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382029 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382032 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382034 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382037 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382039 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382042 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382044 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382047 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382049 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382053 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382056 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382058 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382061 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382063 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382066 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382068 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382071 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:02.392267 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382073 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382076 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382078 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382082 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382084 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382087 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382089 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382092 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382094 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382104 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382107 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382110 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382112 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382115 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382117 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382120 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382122 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382125 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382127 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382130 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:02.392787 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382132 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382135 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382137 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382140 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382144 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382147 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382149 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382153 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382156 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382158 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382161 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382163 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382166 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382169 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382173 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382176 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382180 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382184 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382186 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:02.393295 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382189 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382191 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382194 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382196 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382199 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382203 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:02.382205 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.382260 2570 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.384333 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.385124 2570 server.go:1014] "Starting client certificate rotation" Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.385209 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.385727 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.386815 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 00:23:02.393760 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.389267 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 00:23:02.399310 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.399297 2570 log.go:25] "Validated CRI v1 runtime API" Nov 01 00:23:02.402568 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.402546 2570 log.go:25] "Validated CRI v1 image API" Nov 01 00:23:02.403349 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.403325 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 00:23:02.408003 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.407981 2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7b9b65ff-87d0-4284-b668-2a6f56018008:/dev/nvme0n1p4 d5938dd8-c895-4fb8-8d88-2abf3518cf3c:/dev/nvme0n1p3] Nov 01 00:23:02.408063 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.408003 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 00:23:02.413725 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.413612 2570 manager.go:217] Machine: {Timestamp:2026-11-01 00:23:02.412369643 +0000 UTC m=+0.288981025 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099674 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca34a5227a87bc904da69085e59fa SystemUUID:ec2ca34a-5227-a87b-c904-da69085e59fa BootID:642623d8-4185-403a-82e3-ec92f3e40cc5 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:ce:f7:d0:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:ce:f7:d0:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:ae:eb:73:cc:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 00:23:02.413725 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.413721 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 00:23:02.413862 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.413841 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 00:23:02.415813 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.415788 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 00:23:02.415994 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.415811 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-93.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 00:23:02.416049 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.415996 2570 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 00:23:02.416049 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.416003 2570 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 00:23:02.416049 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.416021 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 00:23:02.416049 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.416036 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 00:23:02.416912 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.416902 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 00:23:02.420663 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.420649 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 00:23:02.421039 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.421027 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 00:23:02.421643 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.421633 2570 kubelet.go:493] "Attempting to sync node with API server" Nov 01 00:23:02.421683 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.421651 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 00:23:02.421683 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.421662 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 00:23:02.421683 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.421676 2570 kubelet.go:405] "Adding apiserver pod source" Nov 01 00:23:02.421759 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.421685 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 00:23:02.422165 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.422154 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 00:23:02.425342 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.425305 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 00:23:02.430015 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.430000 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 00:23:02.430786 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.430772 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 00:23:02.430825 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.430805 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 00:23:02.430981 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.430969 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 00:23:02.431016 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.430990 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 00:23:02.431016 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.430997 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 00:23:02.431016 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431002 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 00:23:02.431016 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431008 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 00:23:02.431016 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431014 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 00:23:02.431472 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431464 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 00:23:02.431472 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431473 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 00:23:02.431526 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431480 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 00:23:02.431526 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431486 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 00:23:02.431526 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431495 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 00:23:02.431526 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.431504 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 00:23:02.432595 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.432585 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 00:23:02.432648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.432598 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 00:23:02.436519 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.436499 2570 server.go:1263] "Started kubelet" Nov 01 00:23:02.436706 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.436660 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 00:23:02.436770 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.436724 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 00:23:02.437003 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.436970 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 00:23:02.437430 ip-10-0-141-93 systemd[1]: Started Kubernetes Kubelet. Nov 01 00:23:02.437931 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.437912 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 00:23:02.438943 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.438929 2570 server.go:317] "Adding debug handlers to kubelet server" Nov 01 00:23:02.443635 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.443615 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 00:23:02.443886 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.443861 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-93.us-west-1.compute.internal\" not found" Nov 01 00:23:02.443977 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.443624 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 00:23:02.444135 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.444117 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 00:23:02.444485 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444468 2570 factory.go:55] Registering systemd factory Nov 01 00:23:02.444569 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444489 2570 factory.go:223] Registration of the systemd container factory successfully Nov 01 00:23:02.444715 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444703 2570 factory.go:153] Registering CRI-O factory Nov 01 00:23:02.444831 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444719 2570 factory.go:223] Registration of the crio container factory successfully Nov 01 00:23:02.444831 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444767 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 00:23:02.444831 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444792 2570 factory.go:103] Registering Raw factory Nov 01 00:23:02.444831 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444809 2570 manager.go:1199] Started watching for new ooms in manager Nov 01 00:23:02.444964 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444831 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 00:23:02.445013 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.444998 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 00:23:02.445064 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.445015 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 00:23:02.445150 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.445139 2570 reconstruct.go:97] "Volume reconstruction finished" Nov 01 00:23:02.445195 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.445151 2570 reconciler.go:29] "Reconciler: start to sync state" Nov 01 00:23:02.445743 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.445555 2570 manager.go:319] Starting recovery of all containers Nov 01 00:23:02.457263 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.457248 2570 manager.go:324] Recovery completed Nov 01 00:23:02.461567 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.461553 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:02.463530 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.463501 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:02.463578 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.463552 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:02.463578 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.463563 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:02.464085 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.464072 2570 cpu_manager.go:226] "Starting" policy="none" Nov 01 00:23:02.464136 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.464087 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 00:23:02.464136 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.464108 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 00:23:02.464136 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.464111 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:02.464958 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.464944 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:02.465023 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.464975 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:02.465023 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.464990 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:02.466541 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.466527 2570 policy_none.go:50] "Start" Nov 01 00:23:02.466541 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.466543 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 00:23:02.466615 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.466556 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 00:23:02.468620 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.468609 2570 policy_none.go:44] "Start" Nov 01 00:23:02.503977 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.503935 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 00:23:02.506391 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.506375 2570 manager.go:341] "Starting Device Plugin manager" Nov 01 00:23:02.506489 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.506412 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 00:23:02.506489 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.506425 2570 server.go:87] "Starting device plugin registration server" Nov 01 00:23:02.506695 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.506682 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 00:23:02.506724 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.506707 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 00:23:02.506801 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.506785 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 00:23:02.506892 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.506877 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 00:23:02.506892 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.506891 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 00:23:02.507476 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.507457 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 00:23:02.507576 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.507521 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-93.us-west-1.compute.internal\" not found" Nov 01 00:23:02.584263 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.584232 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 00:23:02.585734 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.585713 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 00:23:02.586623 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.586605 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 00:23:02.586716 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.586637 2570 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 00:23:02.586716 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.586672 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 00:23:02.607840 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.607815 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:02.608902 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.608885 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:02.608989 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.608921 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:02.608989 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.608937 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:02.608989 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.608980 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.687826 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.687672 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal"] Nov 01 00:23:02.687826 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.687791 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:02.689398 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.689375 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:02.689529 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.689407 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:02.689529 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.689419 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:02.691635 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.691622 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:02.691823 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.691805 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.691867 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.691838 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:02.692529 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.692512 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:02.692629 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.692544 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:02.692629 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.692559 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:02.692629 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.692513 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:02.692629 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.692625 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:02.692846 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.692636 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:02.694833 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.694815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.694920 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.694851 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:02.695596 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.695578 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:02.695686 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.695603 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:02.695686 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.695615 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:02.717869 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.717842 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 00:23:02.717969 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.717901 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p84nl" Nov 01 00:23:02.730007 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.729985 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-93.us-west-1.compute.internal\" not found" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.732908 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.732889 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-93.us-west-1.compute.internal\" not found" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.744555 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.744533 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-93.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 00:23:02.745236 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.744432 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-93.us-west-1.compute.internal.18e3c4005b6c59fc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-93.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-93.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 00:23:02.436461052 +0000 UTC m=+0.313072428,LastTimestamp:2026-11-01 00:23:02.436461052 +0000 UTC m=+0.313072428,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-93.us-west-1.compute.internal,}" Nov 01 00:23:02.746768 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.746750 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.746830 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.746776 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.746830 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.746793 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ff4fc49ea0c4ad7e09012148b2302a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"57ff4fc49ea0c4ad7e09012148b2302a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.793533 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:02.793496 2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-93.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 00:23:02.847257 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.847225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ff4fc49ea0c4ad7e09012148b2302a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"57ff4fc49ea0c4ad7e09012148b2302a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.847257 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.847258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.847452 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.847277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.847452 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.847372 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ff4fc49ea0c4ad7e09012148b2302a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"57ff4fc49ea0c4ad7e09012148b2302a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.847531 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.847453 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.847531 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.847481 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:02.904705 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.904672 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:02.905448 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.905415 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:02.906019 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:02.906004 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:03.004042 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.003948 2570 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-93.us-west-1.compute.internal\" not found" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.005169 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.005154 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:03.033137 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.033087 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.036231 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.036214 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.098176 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.098130 2570 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-93.us-west-1.compute.internal\" not found" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.204233 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.204198 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:03.205193 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.205178 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:03.205249 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.205210 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:03.205249 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.205221 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:03.205328 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.205255 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.262238 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.262171 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.262238 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.262197 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-93.us-west-1.compute.internal\": node \"ip-10-0-141-93.us-west-1.compute.internal\" not found" Nov 01 00:23:03.343990 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.343947 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.384876 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.384851 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 00:23:03.385431 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.384997 2570 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://abbe3b91050ab48e2ac008ae67e6ea89-9261093490b13aca.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.141.93:48492->3.12.131.48:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.385431 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.385028 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:23:03.385431 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.385044 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 00:23:03.385431 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.385070 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 00:23:03.385431 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.385023 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 00:23:03.385431 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.385084 2570 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://abbe3b91050ab48e2ac008ae67e6ea89-9261093490b13aca.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events\": read tcp 10.0.141.93:48492->3.12.131.48:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-93.us-west-1.compute.internal.18e3c4005fbc817c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-93.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-141-93.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 00:23:02.508822908 +0000 UTC m=+0.385434270,LastTimestamp:2026-11-01 00:23:02.508822908 +0000 UTC m=+0.385434270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-93.us-west-1.compute.internal,}" Nov 01 00:23:03.422934 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.422908 2570 apiserver.go:52] "Watching apiserver" Nov 01 00:23:03.444303 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.444258 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 00:23:03.694269 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.694231 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:23:03.694463 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.694333 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 00:23:03.694726 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.694702 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-qvsc4","openshift-ovn-kubernetes/ovnkube-node-q2d7g","kube-system/global-pull-secret-syncer-x4r6x","kube-system/konnectivity-agent-2j99d","openshift-network-diagnostics/network-check-target-9nfbp","openshift-network-operator/iptables-alerter-d74nd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-w9d7m","openshift-multus/multus-j5dgf","openshift-multus/network-metrics-daemon-v2mlf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq","openshift-cluster-node-tuning-operator/tuned-hpcm2","openshift-dns/node-resolver-f994g"] Nov 01 00:23:03.697220 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.697204 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.699404 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.699381 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.701527 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.701508 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.701605 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.701583 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:03.703619 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.703602 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:03.705651 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.705638 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:03.705712 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.705690 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:03.707765 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.707751 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.709981 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.709962 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.712121 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.712102 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.714371 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.714345 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:03.714458 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.714413 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:03.716688 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.716670 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.718893 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.718878 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.721827 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.721805 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal"] Nov 01 00:23:03.721900 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.721873 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.723198 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.723154 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 00:18:02 +0000 UTC" deadline="2028-05-12 14:21:55.943448131 +0000 UTC" Nov 01 00:23:03.723255 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.723197 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13405h58m52.220253074s" Nov 01 00:23:03.746833 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.746810 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 00:23:03.752302 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752263 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-var-lib-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752434 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752315 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-cnibin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.752434 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752337 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-k8s-cni-cncf-io\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.752434 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-cnibin\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.752434 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.752434 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-systemd-units\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752434 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752434 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752432 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-cni-binary-copy\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752486 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-host\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752501 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-registration-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-env-overrides\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752532 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsz6b\" (UniqueName: \"kubernetes.io/projected/d19fab29-f58d-4668-b635-596a496154ed-kube-api-access-tsz6b\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752554 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-systemd\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ea826e-c81c-4429-9a5d-131a2113fcb3-host\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752592 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-kubelet-config\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.752637 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-netns\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752642 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-kubernetes\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752665 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-tmp\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752715 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-ovn\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-hosts-file\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752887 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcbn\" (UniqueName: \"kubernetes.io/projected/15c3c93f-5487-4bbe-a237-4528983d8d38-kube-api-access-jkcbn\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-log-socket\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.752954 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.752952 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-sys\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753009 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-lib-modules\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753049 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-tuned\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-device-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zvl\" (UniqueName: \"kubernetes.io/projected/3e567114-6e47-4337-9776-1ec7f47be0b8-kube-api-access-29zvl\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753220 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-multus\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-etc-kubernetes\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753337 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-modprobe-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753362 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-conf\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-socket-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753441 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-run\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753448 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 00:23:03.753462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-var-lib-kubelet\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-kubelet\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753498 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-bin\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753519 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-host-slash\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753541 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55540c7-35bd-476f-923d-d22e528e3545-ovn-node-metrics-cert\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753549 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753562 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cx8lz\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkjh\" (UniqueName: \"kubernetes.io/projected/e55540c7-35bd-476f-923d-d22e528e3545-kube-api-access-vqkjh\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753589 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753613 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753600 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-dbus\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753641 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753599 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nz8nc\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753644 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80ea826e-c81c-4429-9a5d-131a2113fcb3-serviceca\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753632 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753617 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753711 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxw4\" (UniqueName: \"kubernetes.io/projected/80ea826e-c81c-4429-9a5d-131a2113fcb3-kube-api-access-pkxw4\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753751 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753763 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753768 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-config\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753809 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-system-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.754074 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753859 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-conf-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753886 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-multus-certs\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753915 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxlh\" (UniqueName: \"kubernetes.io/projected/6ef87adc-7459-49ed-9810-4482be252165-kube-api-access-cnxlh\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.753976 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-slash\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-systemd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754024 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-multus-daemon-config\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754057 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7x9\" (UniqueName: \"kubernetes.io/projected/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-kube-api-access-xf7x9\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754076 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-system-cni-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754113 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754144 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-sys-fs\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-node-log\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-netd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-socket-dir-parent\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754239 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-netns\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754261 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-bin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.755119 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754320 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-kubelet\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754343 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-hostroot\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754367 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-tmp-dir\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysconfig\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ad3371f-f985-4f80-b26f-d69319f24cac-agent-certs\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3371f-f985-4f80-b26f-d69319f24cac-konnectivity-ca\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754508 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-script-lib\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-etc-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754562 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjmz\" (UniqueName: \"kubernetes.io/projected/bd5dd766-39a4-45c1-84e8-fa86b275f100-kube-api-access-9wjmz\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754638 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9wd9\" (UniqueName: \"kubernetes.io/projected/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-kube-api-access-q9wd9\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.754662 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-iptables-alerter-script\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.755327 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.755613 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 00:23:03.755887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.755696 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m6gx8\"" Nov 01 00:23:03.759473 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.759453 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 00:23:03.846345 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.846321 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 00:23:03.846461 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.846394 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mdj7n" Nov 01 00:23:03.855508 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.855634 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.855634 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855534 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-ovn\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.855634 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855556 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.855634 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855585 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.855634 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-hosts-file\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.855634 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855607 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-ovn\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.855634 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855619 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855636 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-hosts-file\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855637 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcbn\" (UniqueName: \"kubernetes.io/projected/15c3c93f-5487-4bbe-a237-4528983d8d38-kube-api-access-jkcbn\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-log-socket\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855690 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-log-socket\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855742 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-sys\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855690 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-sys\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-lib-modules\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-tuned\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-device-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zvl\" (UniqueName: \"kubernetes.io/projected/3e567114-6e47-4337-9776-1ec7f47be0b8-kube-api-access-29zvl\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.855858 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855868 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-multus\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-etc-kubernetes\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-modprobe-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855939 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-conf\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855930 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-etc-kubernetes\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-socket-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855877 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-device-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.855945 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-lib-modules\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-run\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856021 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-modprobe-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856044 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-var-lib-kubelet\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856058 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-run\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-kubelet\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856081 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-var-lib-kubelet\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856091 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-conf\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-bin\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856113 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-kubelet\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.856394 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-host-slash\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856142 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-bin\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55540c7-35bd-476f-923d-d22e528e3545-ovn-node-metrics-cert\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkjh\" (UniqueName: \"kubernetes.io/projected/e55540c7-35bd-476f-923d-d22e528e3545-kube-api-access-vqkjh\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856183 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-host-slash\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856190 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-socket-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-dbus\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856220 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80ea826e-c81c-4429-9a5d-131a2113fcb3-serviceca\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxw4\" (UniqueName: \"kubernetes.io/projected/80ea826e-c81c-4429-9a5d-131a2113fcb3-kube-api-access-pkxw4\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856303 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-dbus\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-config\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-system-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856358 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-conf-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-multus-certs\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.856389 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856408 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxlh\" (UniqueName: \"kubernetes.io/projected/6ef87adc-7459-49ed-9810-4482be252165-kube-api-access-cnxlh\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.857141 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856426 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.856490 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs podName:15c3c93f-5487-4bbe-a237-4528983d8d38 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:04.356440009 +0000 UTC m=+2.233051390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs") pod "network-metrics-daemon-v2mlf" (UID: "15c3c93f-5487-4bbe-a237-4528983d8d38") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856488 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-system-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856497 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-multus-certs\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856461 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-conf-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856521 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-slash\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-systemd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-slash\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-multus-daemon-config\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856582 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-systemd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7x9\" (UniqueName: \"kubernetes.io/projected/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-kube-api-access-xf7x9\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856616 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-system-cni-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856620 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856651 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-system-cni-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856682 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-multus\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856748 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80ea826e-c81c-4429-9a5d-131a2113fcb3-serviceca\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.858032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-sys-fs\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-node-log\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856827 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-netd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856840 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856844 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-socket-dir-parent\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856843 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-sys-fs\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856871 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-node-log\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856892 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-socket-dir-parent\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856899 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-netd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-netns\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-bin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856965 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-netns\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-kubelet\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857018 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-kubelet\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-hostroot\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.856998 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-config\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857043 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-hostroot\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857008 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-bin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.858864 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857051 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-tmp-dir\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysconfig\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857101 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ad3371f-f985-4f80-b26f-d69319f24cac-agent-certs\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3371f-f985-4f80-b26f-d69319f24cac-konnectivity-ca\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-script-lib\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857207 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-etc-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857247 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjmz\" (UniqueName: \"kubernetes.io/projected/bd5dd766-39a4-45c1-84e8-fa86b275f100-kube-api-access-9wjmz\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.857338 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857370 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-tmp-dir\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.857395 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret podName:5f953789-9f33-475b-9687-714c6960973c nodeName:}" failed. No retries permitted until 2026-11-01 00:23:04.357377256 +0000 UTC m=+2.233988637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret") pod "global-pull-secret-syncer-x4r6x" (UID: "5f953789-9f33-475b-9687-714c6960973c") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857410 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-etc-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857422 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysconfig\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.859648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9wd9\" (UniqueName: \"kubernetes.io/projected/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-kube-api-access-q9wd9\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857459 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-iptables-alerter-script\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857494 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857507 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-var-lib-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-cnibin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-k8s-cni-cncf-io\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-cnibin\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857608 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-systemd-units\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857660 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857680 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-script-lib\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-cni-binary-copy\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857722 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-host\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857726 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-registration-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-env-overrides\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsz6b\" (UniqueName: \"kubernetes.io/projected/d19fab29-f58d-4668-b635-596a496154ed-kube-api-access-tsz6b\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.860409 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857834 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3371f-f985-4f80-b26f-d69319f24cac-konnectivity-ca\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857838 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-systemd\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857871 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-systemd-units\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857877 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-systemd\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ea826e-c81c-4429-9a5d-131a2113fcb3-host\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857905 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-kubelet-config\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857946 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857947 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-netns\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.857998 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-netns\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858021 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ea826e-c81c-4429-9a5d-131a2113fcb3-host\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858053 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-registration-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858083 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-host\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-kubernetes\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858144 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-tmp\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858162 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858699 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-kubelet-config\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858752 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-kubernetes\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858874 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6c54w\"" Nov 01 00:23:03.861147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.858941 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hhlh7\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.859172 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.859572 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-env-overrides\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.859577 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.859821 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.859909 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.860341 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rhkgk\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.860376 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.860762 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.860939 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-var-lib-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861062 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-cnibin\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861228 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861262 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861465 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861658 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4lrql\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861683 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-iptables-alerter-script\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861726 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k669s\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861229 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55540c7-35bd-476f-923d-d22e528e3545-ovn-node-metrics-cert\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861766 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861883 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zctdk\"" Nov 01 00:23:03.862072 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.861941 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 00:23:03.862972 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.862245 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-k8s-cni-cncf-io\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.862972 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.862333 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-tuned\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.862972 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.862340 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-cnibin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.862972 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.862748 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.862972 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.862766 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ad3371f-f985-4f80-b26f-d69319f24cac-agent-certs\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:03.863201 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.863107 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-tmp\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.863368 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.863342 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 00:23:03.866621 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.866600 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.867505 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.867483 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-multus-daemon-config\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.868382 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.868366 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.868435 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.868382 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-cni-binary-copy\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:03.914510 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.914207 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:23:03.914864 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.914521 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:23:03.914864 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.914539 2570 projected.go:196] Error preparing data for projected volume kube-api-access-495mv for pod openshift-network-diagnostics/network-check-target-9nfbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:03.914864 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:03.914648 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv podName:bf719590-ce53-4a44-af67-05c687f68258 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:04.414625595 +0000 UTC m=+2.291236990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-495mv" (UniqueName: "kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv") pod "network-check-target-9nfbp" (UID: "bf719590-ce53-4a44-af67-05c687f68258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:03.916750 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.916722 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcbn\" (UniqueName: \"kubernetes.io/projected/15c3c93f-5487-4bbe-a237-4528983d8d38-kube-api-access-jkcbn\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:03.916860 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.916724 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zvl\" (UniqueName: \"kubernetes.io/projected/3e567114-6e47-4337-9776-1ec7f47be0b8-kube-api-access-29zvl\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:03.917145 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.917125 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkjh\" (UniqueName: \"kubernetes.io/projected/e55540c7-35bd-476f-923d-d22e528e3545-kube-api-access-vqkjh\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:03.917398 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.917369 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7x9\" (UniqueName: \"kubernetes.io/projected/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-kube-api-access-xf7x9\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:03.917476 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.917402 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxlh\" (UniqueName: \"kubernetes.io/projected/6ef87adc-7459-49ed-9810-4482be252165-kube-api-access-cnxlh\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:03.917931 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.917906 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxw4\" (UniqueName: \"kubernetes.io/projected/80ea826e-c81c-4429-9a5d-131a2113fcb3-kube-api-access-pkxw4\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:03.922366 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.922345 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9wd9\" (UniqueName: \"kubernetes.io/projected/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-kube-api-access-q9wd9\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:03.922557 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.922538 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjmz\" (UniqueName: \"kubernetes.io/projected/bd5dd766-39a4-45c1-84e8-fa86b275f100-kube-api-access-9wjmz\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:03.922618 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:03.922597 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsz6b\" (UniqueName: \"kubernetes.io/projected/d19fab29-f58d-4668-b635-596a496154ed-kube-api-access-tsz6b\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:23:04.033086 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.033063 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:23:04.063559 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.063525 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:04.068943 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.068921 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:04.075438 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.075417 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:23:04.098345 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.098316 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55540c7_35bd_476f_923d_d22e528e3545.slice/crio-74298718edc986141ebc52073eb6b38b8b6600d8339dcb581f2fa20f2640a91a WatchSource:0}: Error finding container 74298718edc986141ebc52073eb6b38b8b6600d8339dcb581f2fa20f2640a91a: Status 404 returned error can't find the container with id 74298718edc986141ebc52073eb6b38b8b6600d8339dcb581f2fa20f2640a91a Nov 01 00:23:04.099019 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.098989 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad3371f_f985_4f80_b26f_d69319f24cac.slice/crio-e52c8c3ee9469ec090cc1711c535529904b8edf292ca17580df2f784d2ddcfa4 WatchSource:0}: Error finding container e52c8c3ee9469ec090cc1711c535529904b8edf292ca17580df2f784d2ddcfa4: Status 404 returned error can't find the container with id e52c8c3ee9469ec090cc1711c535529904b8edf292ca17580df2f784d2ddcfa4 Nov 01 00:23:04.099894 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.099867 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ff4fc49ea0c4ad7e09012148b2302a.slice/crio-d568f7d494cf16f683795b12308fa36422f85d82cd5de081c8ae4cdfa2139186 WatchSource:0}: Error finding container d568f7d494cf16f683795b12308fa36422f85d82cd5de081c8ae4cdfa2139186: Status 404 returned error can't find the container with id d568f7d494cf16f683795b12308fa36422f85d82cd5de081c8ae4cdfa2139186 Nov 01 00:23:04.103474 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.103440 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f77eb99_e6b3_4ed7_a9e0_a5667e93807f.slice/crio-4e63241786a261f1ca96741f564820e6fa79be6772a9bded988e9610598ea232 WatchSource:0}: Error finding container 4e63241786a261f1ca96741f564820e6fa79be6772a9bded988e9610598ea232: Status 404 returned error can't find the container with id 4e63241786a261f1ca96741f564820e6fa79be6772a9bded988e9610598ea232 Nov 01 00:23:04.104609 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.104489 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:23:04.104760 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.104722 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74b1dfaae23b56af399f563bf8f80d7.slice/crio-67d02bfe87504573f9473b10cb83becef096897d9004e7553b8145d0934b5bdb WatchSource:0}: Error finding container 67d02bfe87504573f9473b10cb83becef096897d9004e7553b8145d0934b5bdb: Status 404 returned error can't find the container with id 67d02bfe87504573f9473b10cb83becef096897d9004e7553b8145d0934b5bdb Nov 01 00:23:04.104819 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.104785 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:23:04.113406 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.113377 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5dgf" Nov 01 00:23:04.113600 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.113583 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef87adc_7459_49ed_9810_4482be252165.slice/crio-4c45e0d057b00b805e16c4cb0878a08fe02ccdc83b6f33e2dda0d79876a404ea WatchSource:0}: Error finding container 4c45e0d057b00b805e16c4cb0878a08fe02ccdc83b6f33e2dda0d79876a404ea: Status 404 returned error can't find the container with id 4c45e0d057b00b805e16c4cb0878a08fe02ccdc83b6f33e2dda0d79876a404ea Nov 01 00:23:04.121897 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.121877 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:23:04.122190 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.122169 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19fab29_f58d_4668_b635_596a496154ed.slice/crio-611b04993ccdb245bd8758384f55c870ccaaf227d5515750d08a34238b55db7c WatchSource:0}: Error finding container 611b04993ccdb245bd8758384f55c870ccaaf227d5515750d08a34238b55db7c: Status 404 returned error can't find the container with id 611b04993ccdb245bd8758384f55c870ccaaf227d5515750d08a34238b55db7c Nov 01 00:23:04.130506 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.130486 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e567114_6e47_4337_9776_1ec7f47be0b8.slice/crio-670484c9353bc5993f26641d6532bc27a214a32ef08aaef251fbda58cfc2bc80 WatchSource:0}: Error finding container 670484c9353bc5993f26641d6532bc27a214a32ef08aaef251fbda58cfc2bc80: Status 404 returned error can't find the container with id 670484c9353bc5993f26641d6532bc27a214a32ef08aaef251fbda58cfc2bc80 Nov 01 00:23:04.141722 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.141701 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:23:04.146293 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.146250 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f994g" Nov 01 00:23:04.150930 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.150902 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5dd766_39a4_45c1_84e8_fa86b275f100.slice/crio-f1164141c514c16aa79a12dd7effccab396a630d952d2a88e31f8135f29d27bb WatchSource:0}: Error finding container f1164141c514c16aa79a12dd7effccab396a630d952d2a88e31f8135f29d27bb: Status 404 returned error can't find the container with id f1164141c514c16aa79a12dd7effccab396a630d952d2a88e31f8135f29d27bb Nov 01 00:23:04.155149 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.155127 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5c470e_0673_4c56_a4e8_feb9fa32b67e.slice/crio-4dce003717d516c671a1c0a1099a02b3d5547c8d88c03545c168e3156a658abe WatchSource:0}: Error finding container 4dce003717d516c671a1c0a1099a02b3d5547c8d88c03545c168e3156a658abe: Status 404 returned error can't find the container with id 4dce003717d516c671a1c0a1099a02b3d5547c8d88c03545c168e3156a658abe Nov 01 00:23:04.362212 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.362119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:04.362405 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.362209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:04.362405 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:04.362276 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:04.362405 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:04.362337 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:04.362405 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:04.362357 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret podName:5f953789-9f33-475b-9687-714c6960973c nodeName:}" failed. No retries permitted until 2026-11-01 00:23:05.362342992 +0000 UTC m=+3.238954371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret") pod "global-pull-secret-syncer-x4r6x" (UID: "5f953789-9f33-475b-9687-714c6960973c") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:04.362405 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:04.362387 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs podName:15c3c93f-5487-4bbe-a237-4528983d8d38 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:05.362372479 +0000 UTC m=+3.238983843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs") pod "network-metrics-daemon-v2mlf" (UID: "15c3c93f-5487-4bbe-a237-4528983d8d38") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:04.463421 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.463391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:04.463804 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:04.463519 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:23:04.463804 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:04.463534 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:23:04.463804 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:04.463544 2570 projected.go:196] Error preparing data for projected volume kube-api-access-495mv for pod openshift-network-diagnostics/network-check-target-9nfbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:04.463804 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:04.463589 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv podName:bf719590-ce53-4a44-af67-05c687f68258 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:05.463576281 +0000 UTC m=+3.340187644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-495mv" (UniqueName: "kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv") pod "network-check-target-9nfbp" (UID: "bf719590-ce53-4a44-af67-05c687f68258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:04.593163 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.593113 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"57ff4fc49ea0c4ad7e09012148b2302a","Type":"ContainerStarted","Data":"d568f7d494cf16f683795b12308fa36422f85d82cd5de081c8ae4cdfa2139186"} Nov 01 00:23:04.594198 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.594178 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"74298718edc986141ebc52073eb6b38b8b6600d8339dcb581f2fa20f2640a91a"} Nov 01 00:23:04.595130 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.595103 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" event={"ID":"3e567114-6e47-4337-9776-1ec7f47be0b8","Type":"ContainerStarted","Data":"670484c9353bc5993f26641d6532bc27a214a32ef08aaef251fbda58cfc2bc80"} Nov 01 00:23:04.596077 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.596056 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d74nd" event={"ID":"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f","Type":"ContainerStarted","Data":"4e63241786a261f1ca96741f564820e6fa79be6772a9bded988e9610598ea232"} Nov 01 00:23:04.597039 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.597013 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f994g" event={"ID":"fa5c470e-0673-4c56-a4e8-feb9fa32b67e","Type":"ContainerStarted","Data":"4dce003717d516c671a1c0a1099a02b3d5547c8d88c03545c168e3156a658abe"} Nov 01 00:23:04.598027 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.598007 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" event={"ID":"bd5dd766-39a4-45c1-84e8-fa86b275f100","Type":"ContainerStarted","Data":"f1164141c514c16aa79a12dd7effccab396a630d952d2a88e31f8135f29d27bb"} Nov 01 00:23:04.599008 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.598990 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5dgf" event={"ID":"d19fab29-f58d-4668-b635-596a496154ed","Type":"ContainerStarted","Data":"611b04993ccdb245bd8758384f55c870ccaaf227d5515750d08a34238b55db7c"} Nov 01 00:23:04.599929 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.599913 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" event={"ID":"6ef87adc-7459-49ed-9810-4482be252165","Type":"ContainerStarted","Data":"4c45e0d057b00b805e16c4cb0878a08fe02ccdc83b6f33e2dda0d79876a404ea"} Nov 01 00:23:04.600828 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.600812 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"a74b1dfaae23b56af399f563bf8f80d7","Type":"ContainerStarted","Data":"67d02bfe87504573f9473b10cb83becef096897d9004e7553b8145d0934b5bdb"} Nov 01 00:23:04.601674 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.601657 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2j99d" event={"ID":"7ad3371f-f985-4f80-b26f-d69319f24cac","Type":"ContainerStarted","Data":"e52c8c3ee9469ec090cc1711c535529904b8edf292ca17580df2f784d2ddcfa4"} Nov 01 00:23:04.651378 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.651305 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:04.798144 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:04.798116 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ea826e_c81c_4429_9a5d_131a2113fcb3.slice/crio-1234b17a183cb5348a6d0f5a0113df6fae4857630b14d99e244027db584cca67 WatchSource:0}: Error finding container 1234b17a183cb5348a6d0f5a0113df6fae4857630b14d99e244027db584cca67: Status 404 returned error can't find the container with id 1234b17a183cb5348a6d0f5a0113df6fae4857630b14d99e244027db584cca67 Nov 01 00:23:04.844103 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.844072 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:04.849137 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.849095 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:18:03 +0000 UTC" deadline="2028-05-04 02:11:21.927270739 +0000 UTC" Nov 01 00:23:04.849137 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:04.849134 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13201h48m17.07814018s" Nov 01 00:23:05.015454 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.015383 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:05.371873 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.371827 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:05.372091 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.371917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:05.372091 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.372045 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:05.372215 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.372106 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs podName:15c3c93f-5487-4bbe-a237-4528983d8d38 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:07.372089085 +0000 UTC m=+5.248700449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs") pod "network-metrics-daemon-v2mlf" (UID: "15c3c93f-5487-4bbe-a237-4528983d8d38") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:05.372215 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.372166 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:05.372215 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.372199 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret podName:5f953789-9f33-475b-9687-714c6960973c nodeName:}" failed. No retries permitted until 2026-11-01 00:23:07.372187193 +0000 UTC m=+5.248798559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret") pod "global-pull-secret-syncer-x4r6x" (UID: "5f953789-9f33-475b-9687-714c6960973c") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:05.472555 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.472516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:05.473040 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.472697 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:23:05.473040 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.472718 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:23:05.473040 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.472730 2570 projected.go:196] Error preparing data for projected volume kube-api-access-495mv for pod openshift-network-diagnostics/network-check-target-9nfbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:05.473040 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.472785 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv podName:bf719590-ce53-4a44-af67-05c687f68258 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:07.47276879 +0000 UTC m=+5.349380177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-495mv" (UniqueName: "kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv") pod "network-check-target-9nfbp" (UID: "bf719590-ce53-4a44-af67-05c687f68258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:05.587869 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.587836 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:05.588110 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.587964 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:05.588110 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.588060 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:05.588335 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.588148 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:05.588335 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.588211 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:05.588335 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:05.588301 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:05.622354 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.622251 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qvsc4" event={"ID":"80ea826e-c81c-4429-9a5d-131a2113fcb3","Type":"ContainerStarted","Data":"1234b17a183cb5348a6d0f5a0113df6fae4857630b14d99e244027db584cca67"} Nov 01 00:23:05.850244 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.850193 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:18:03 +0000 UTC" deadline="2028-07-18 07:45:53.329474452 +0000 UTC" Nov 01 00:23:05.850244 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:05.850242 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15007h22m47.479237502s" Nov 01 00:23:07.391575 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:07.391262 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:07.391575 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:07.391326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:07.391575 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.391441 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:07.391575 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.391488 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret podName:5f953789-9f33-475b-9687-714c6960973c nodeName:}" failed. No retries permitted until 2026-11-01 00:23:11.391476153 +0000 UTC m=+9.268087517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret") pod "global-pull-secret-syncer-x4r6x" (UID: "5f953789-9f33-475b-9687-714c6960973c") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:07.391575 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.391533 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:07.391575 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.391554 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs podName:15c3c93f-5487-4bbe-a237-4528983d8d38 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:11.39154801 +0000 UTC m=+9.268159373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs") pod "network-metrics-daemon-v2mlf" (UID: "15c3c93f-5487-4bbe-a237-4528983d8d38") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:07.494209 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:07.494132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:07.499720 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.495726 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:23:07.499720 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.495753 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:23:07.499720 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.496239 2570 projected.go:196] Error preparing data for projected volume kube-api-access-495mv for pod openshift-network-diagnostics/network-check-target-9nfbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:07.499720 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.496345 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv podName:bf719590-ce53-4a44-af67-05c687f68258 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:11.496316723 +0000 UTC m=+9.372928104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-495mv" (UniqueName: "kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv") pod "network-check-target-9nfbp" (UID: "bf719590-ce53-4a44-af67-05c687f68258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:07.587861 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:07.587819 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:07.588047 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.587967 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:07.588047 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:07.588046 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:07.588151 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.588122 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:07.588216 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:07.588174 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:07.588272 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:07.588240 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:08.643211 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:08.643173 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="17889bfe872216eac0538c2d585ba5dbd11ae78812c8741a9f1e4d311cd3bcb3" exitCode=0 Nov 01 00:23:08.643900 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:08.643261 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" event={"ID":"6ef87adc-7459-49ed-9810-4482be252165","Type":"ContainerDied","Data":"17889bfe872216eac0538c2d585ba5dbd11ae78812c8741a9f1e4d311cd3bcb3"} Nov 01 00:23:08.650987 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:08.650758 2570 generic.go:356] "Generic (PLEG): container finished" podID="a74b1dfaae23b56af399f563bf8f80d7" containerID="39a18f4b9b2d0535b7a8eae2a9a4d31ac1a67eb99547f89909fecd77a230645f" exitCode=0 Nov 01 00:23:08.651212 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:08.651009 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"a74b1dfaae23b56af399f563bf8f80d7","Type":"ContainerDied","Data":"39a18f4b9b2d0535b7a8eae2a9a4d31ac1a67eb99547f89909fecd77a230645f"} Nov 01 00:23:09.586950 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:09.586911 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:09.587131 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:09.587031 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:09.587131 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:09.587048 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:09.587131 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:09.587125 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:09.587311 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:09.587163 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:09.587311 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:09.587226 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:09.661078 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:09.661035 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"a74b1dfaae23b56af399f563bf8f80d7","Type":"ContainerStarted","Data":"dd19dc94f53f43d860fb31b855e6b322530f363909e38ee273a4a9b624c89faf"} Nov 01 00:23:09.664654 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:09.664622 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2j99d" event={"ID":"7ad3371f-f985-4f80-b26f-d69319f24cac","Type":"ContainerStarted","Data":"d89a3f44d1e4d84a3a92927c62ee7467e6f142f2ef40800ca9c4bf0ee4aa556b"} Nov 01 00:23:09.682870 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:09.682831 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"57ff4fc49ea0c4ad7e09012148b2302a","Type":"ContainerStarted","Data":"5f551eca9a3d02b351bd948310aadb47a9f285ac77db5e1ced21ecd9f4a34dfb"} Nov 01 00:23:09.895510 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:09.895404 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" podStartSLOduration=6.895386748 podStartE2EDuration="6.895386748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:09.782597225 +0000 UTC m=+7.659208611" watchObservedRunningTime="2026-11-01 00:23:09.895386748 +0000 UTC m=+7.771998135" Nov 01 00:23:10.010771 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:10.010601 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" podStartSLOduration=7.01058543 podStartE2EDuration="7.01058543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:09.896520176 +0000 UTC m=+7.773131563" watchObservedRunningTime="2026-11-01 00:23:10.01058543 +0000 UTC m=+7.887196814" Nov 01 00:23:10.690277 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:10.690245 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" event={"ID":"3e567114-6e47-4337-9776-1ec7f47be0b8","Type":"ContainerStarted","Data":"c95a871abb4efb7ba024f59263180159e8f7aa2b6bd899cd6f5bca9c0a598b07"} Nov 01 00:23:11.427926 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.427101 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:11.427926 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.427167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:11.427926 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.427351 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:11.427926 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.427405 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret podName:5f953789-9f33-475b-9687-714c6960973c nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.427387738 +0000 UTC m=+17.303999116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret") pod "global-pull-secret-syncer-x4r6x" (UID: "5f953789-9f33-475b-9687-714c6960973c") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:11.427926 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.427775 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:11.427926 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.427821 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs podName:15c3c93f-5487-4bbe-a237-4528983d8d38 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.427804747 +0000 UTC m=+17.304416120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs") pod "network-metrics-daemon-v2mlf" (UID: "15c3c93f-5487-4bbe-a237-4528983d8d38") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:11.528236 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.528088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:11.528421 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.528310 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:23:11.528421 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.528341 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:23:11.528421 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.528354 2570 projected.go:196] Error preparing data for projected volume kube-api-access-495mv for pod openshift-network-diagnostics/network-check-target-9nfbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:11.528421 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.528420 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv podName:bf719590-ce53-4a44-af67-05c687f68258 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.528401259 +0000 UTC m=+17.405012624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-495mv" (UniqueName: "kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv") pod "network-check-target-9nfbp" (UID: "bf719590-ce53-4a44-af67-05c687f68258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:11.587321 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.586882 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:11.587321 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.587010 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:11.587321 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.587090 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:11.587321 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.587158 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:11.587321 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.587209 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:11.587321 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:11.587268 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:11.695030 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.694312 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qvsc4" event={"ID":"80ea826e-c81c-4429-9a5d-131a2113fcb3","Type":"ContainerStarted","Data":"16dc1b82deb0d9b1318b27dc27faa6090bd9d0cc747427b27fb1c8bdf0a19452"} Nov 01 00:23:11.808927 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.808862 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2j99d" podStartSLOduration=3.729227147 podStartE2EDuration="8.808842846s" totalImagesPullingTime="5.079615699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:10.011228188 +0000 UTC m=+7.887839573" watchObservedRunningTime="2026-11-01 00:23:11.808842846 +0000 UTC m=+9.685454235" Nov 01 00:23:11.809715 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:11.809646 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qvsc4" podStartSLOduration=2.796075375 podStartE2EDuration="8.809630501s" totalImagesPullingTime="6.013555126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:11.808394902 +0000 UTC m=+9.685006321" watchObservedRunningTime="2026-11-01 00:23:11.809630501 +0000 UTC m=+9.686241889" Nov 01 00:23:12.802243 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:12.801832 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 00:23:12.813607 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:12.812925 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:12.814196 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:12.813991 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:13.512261 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.512140 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T00:23:12.801860474Z","UUID":"581609fa-a9c9-44d0-b170-d310bb2ecf46","Handler":null,"Name":"","Endpoint":""} Nov 01 00:23:13.513910 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.513891 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 00:23:13.514032 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.513924 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 00:23:13.587163 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.587134 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:13.587363 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:13.587246 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:13.587444 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.587374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:13.587490 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:13.587464 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:13.587529 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.587518 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:13.587611 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:13.587587 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:13.704344 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.704220 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="fc27f404fb62fd39b3a31ddf0264524d4dfb63a8aa10493fc3dcf3c3d419e7f6" exitCode=0 Nov 01 00:23:13.704496 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.704344 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" event={"ID":"6ef87adc-7459-49ed-9810-4482be252165","Type":"ContainerDied","Data":"fc27f404fb62fd39b3a31ddf0264524d4dfb63a8aa10493fc3dcf3c3d419e7f6"} Nov 01 00:23:13.707116 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.707053 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" event={"ID":"3e567114-6e47-4337-9776-1ec7f47be0b8","Type":"ContainerStarted","Data":"641de299c9739dc0f23b60b5ef71fffa64ed3e2869d344ce3d822c10bbc7f94a"} Nov 01 00:23:13.709222 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.709197 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f994g" event={"ID":"fa5c470e-0673-4c56-a4e8-feb9fa32b67e","Type":"ContainerStarted","Data":"381fc24ebebe6b2a14736d0e151822ed0ca88c2af5080a3fa5c54407a293a1d2"} Nov 01 00:23:13.710902 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.710881 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5dgf" event={"ID":"d19fab29-f58d-4668-b635-596a496154ed","Type":"ContainerStarted","Data":"5446ff65e230542df28c28541cfd956ae54be3fa30833102751f681a7ad922d3"} Nov 01 00:23:13.711271 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.711250 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:13.711618 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:13.711599 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:23:14.174351 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:14.174271 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-j5dgf" podStartSLOduration=2.680848347 podStartE2EDuration="11.174251471s" totalImagesPullingTime="8.493403124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:14.060048855 +0000 UTC m=+11.936660241" watchObservedRunningTime="2026-11-01 00:23:14.174251471 +0000 UTC m=+12.050862858" Nov 01 00:23:14.716602 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:14.716564 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" event={"ID":"3e567114-6e47-4337-9776-1ec7f47be0b8","Type":"ContainerStarted","Data":"37bdd7782e2a2d286c3df144a46786ca7c0efaa4ed3bf52267b8c27e252b5ad7"} Nov 01 00:23:14.719646 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:14.719598 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d74nd" event={"ID":"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f","Type":"ContainerStarted","Data":"6011898cfc4138f62481147197792ac1e4a7a29108f698e4c562f040fd43c00c"} Nov 01 00:23:14.833756 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:14.833697 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" podStartSLOduration=2.029587617 podStartE2EDuration="11.833682926s" totalImagesPullingTime="9.804095309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:14.833620024 +0000 UTC m=+12.710231410" watchObservedRunningTime="2026-11-01 00:23:14.833682926 +0000 UTC m=+12.710294310" Nov 01 00:23:14.833943 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:14.833868 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-f994g" podStartSLOduration=3.097652594 podStartE2EDuration="11.833861405s" totalImagesPullingTime="8.736208811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:14.175472878 +0000 UTC m=+12.052084267" watchObservedRunningTime="2026-11-01 00:23:14.833861405 +0000 UTC m=+12.710472790" Nov 01 00:23:15.587330 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:15.587277 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:15.587805 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:15.587370 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:15.587805 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:15.587463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:15.587805 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:15.587474 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:15.587805 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:15.587543 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:15.587805 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:15.587623 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:15.723900 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:15.723866 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="2ebdd7c83a6afd78534db73663d35e965b481a869b62ad02c511d26def630939" exitCode=0 Nov 01 00:23:15.724057 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:15.723946 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" event={"ID":"6ef87adc-7459-49ed-9810-4482be252165","Type":"ContainerDied","Data":"2ebdd7c83a6afd78534db73663d35e965b481a869b62ad02c511d26def630939"} Nov 01 00:23:15.725412 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:15.725388 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" event={"ID":"bd5dd766-39a4-45c1-84e8-fa86b275f100","Type":"ContainerStarted","Data":"5e17327eb9ca998f689609a59e1ad27c80de3f7a2ae67cc31b4863ad9ad40319"} Nov 01 00:23:15.844645 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:15.844546 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d74nd" podStartSLOduration=3.825968318 podStartE2EDuration="12.844531975s" totalImagesPullingTime="9.018563657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:14.951109713 +0000 UTC m=+12.827721099" watchObservedRunningTime="2026-11-01 00:23:15.844531975 +0000 UTC m=+13.721143360" Nov 01 00:23:15.962084 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:15.962017 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" podStartSLOduration=2.051851021 podStartE2EDuration="12.962004156s" totalImagesPullingTime="10.910153135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:15.961393514 +0000 UTC m=+13.838004901" watchObservedRunningTime="2026-11-01 00:23:15.962004156 +0000 UTC m=+13.838615556" Nov 01 00:23:17.587648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:17.587418 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:17.588189 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:17.587446 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:17.588189 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:17.587756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:17.588189 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:17.587490 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:17.588189 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:17.587816 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:17.588189 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:17.587884 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:17.733244 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:17.733128 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="3862dd8a7a43bb68fe7397f32685c496ed02b8c254a667914d0e620cc1b4ea94" exitCode=0 Nov 01 00:23:17.733244 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:17.733182 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" event={"ID":"6ef87adc-7459-49ed-9810-4482be252165","Type":"ContainerDied","Data":"3862dd8a7a43bb68fe7397f32685c496ed02b8c254a667914d0e620cc1b4ea94"} Nov 01 00:23:19.404539 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.404504 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:19.489861 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.489827 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:19.490029 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.489876 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:19.490029 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.489976 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:19.490029 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.490010 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:19.490145 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.490054 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs podName:15c3c93f-5487-4bbe-a237-4528983d8d38 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.49003528 +0000 UTC m=+33.366646656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs") pod "network-metrics-daemon-v2mlf" (UID: "15c3c93f-5487-4bbe-a237-4528983d8d38") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:19.490145 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.490070 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret podName:5f953789-9f33-475b-9687-714c6960973c nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.490063335 +0000 UTC m=+33.366674698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret") pod "global-pull-secret-syncer-x4r6x" (UID: "5f953789-9f33-475b-9687-714c6960973c") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:19.587011 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.586876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:19.587011 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.586876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:19.587129 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.587016 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:19.587129 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.587058 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:19.587213 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.587163 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:19.587254 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.587232 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:19.591308 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.591202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:19.591408 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.591351 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:23:19.591408 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.591372 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:23:19.591408 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.591384 2570 projected.go:196] Error preparing data for projected volume kube-api-access-495mv for pod openshift-network-diagnostics/network-check-target-9nfbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:19.591571 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:19.591436 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv podName:bf719590-ce53-4a44-af67-05c687f68258 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.591420034 +0000 UTC m=+33.468031398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-495mv" (UniqueName: "kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv") pod "network-check-target-9nfbp" (UID: "bf719590-ce53-4a44-af67-05c687f68258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:19.663121 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.662937 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:19.739687 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.739658 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"709cefd88a1c0e5c53cffe1d891ceea05a12b6f7d42cb5baa08bcc498c835fb9"} Nov 01 00:23:19.739825 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.739741 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"051d173fa90ea8f2a0e716716b153696bacb27d4f24bf30c72296cdafcd0bedd"} Nov 01 00:23:19.739825 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:19.739786 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"9a3b1989f2539957a2d4639a740ac3200d97b97fb72285d40427b0558a0d180f"} Nov 01 00:23:20.744650 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:20.744619 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="240bfa78bba9aeede7d28e4a7571f97faac0242bbd458345eefc208a6c510c86" exitCode=0 Nov 01 00:23:20.745372 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:20.744697 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" event={"ID":"6ef87adc-7459-49ed-9810-4482be252165","Type":"ContainerDied","Data":"240bfa78bba9aeede7d28e4a7571f97faac0242bbd458345eefc208a6c510c86"} Nov 01 00:23:20.747980 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:20.747954 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"871d658e4a9b3cb9bdb9a252100d7ff8166c95ac8166c2e59caee4c54e466d73"} Nov 01 00:23:20.748055 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:20.748039 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"9f5e50ab78316a2455ce690e21f2b0c6a973985f503b6fcf624c4f1902047587"} Nov 01 00:23:20.748108 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:20.748096 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"7c80b90bde78c6522563f10f65877e9118556666032531d6cfb1c3e21414bb8f"} Nov 01 00:23:20.842793 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:20.842764 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:21.054988 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:21.054960 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:21.587258 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:21.587221 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:21.587258 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:21.587246 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:21.587431 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:21.587356 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:21.587431 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:21.587379 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:21.587504 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:21.587439 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:21.587540 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:21.587498 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:21.753422 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:21.753380 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="2edc46829be3b0c730b00c503ff3174b03860aaf13c213c93cc5e2d05845bca7" exitCode=0 Nov 01 00:23:21.753422 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:21.753425 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" event={"ID":"6ef87adc-7459-49ed-9810-4482be252165","Type":"ContainerDied","Data":"2edc46829be3b0c730b00c503ff3174b03860aaf13c213c93cc5e2d05845bca7"} Nov 01 00:23:22.758612 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:22.758573 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"72da6312d6a05ff53b5602e1cac86659dbffb29b233d1cd98877d72ca5fdb040"} Nov 01 00:23:22.761862 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:22.761834 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" event={"ID":"6ef87adc-7459-49ed-9810-4482be252165","Type":"ContainerStarted","Data":"283c113e0613b1eab899b509ab2d5d15bac17dc8ee6712b89bc6ba2dd244aa47"} Nov 01 00:23:22.884030 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:22.883980 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w9d7m" podStartSLOduration=7.080975861 podStartE2EDuration="19.883965969s" totalImagesPullingTime="12.802990108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:22.883156111 +0000 UTC m=+20.759767509" watchObservedRunningTime="2026-11-01 00:23:22.883965969 +0000 UTC m=+20.760577354" Nov 01 00:23:23.071034 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:23.070982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:23.285408 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:23.285191 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:23.587087 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:23.587041 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:23.587087 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:23.587071 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:23.587347 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:23.587156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:23.587347 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:23.587168 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:23.587347 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:23.587237 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:23.587347 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:23.587317 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:24.769645 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:24.769609 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" event={"ID":"e55540c7-35bd-476f-923d-d22e528e3545","Type":"ContainerStarted","Data":"9c53a8f5f8b4c99d587538c664739c8fa342704727e5b3d2bafae4f66379689f"} Nov 01 00:23:24.770227 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:24.769980 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:24.770227 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:24.770071 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:24.785711 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:24.785687 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:25.047558 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:25.047504 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" podStartSLOduration=6.713464104 podStartE2EDuration="22.04749133s" totalImagesPullingTime="15.334027226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:25.046775389 +0000 UTC m=+22.923386773" watchObservedRunningTime="2026-11-01 00:23:25.04749133 +0000 UTC m=+22.924102715" Nov 01 00:23:25.587925 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:25.587880 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:25.587925 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:25.587908 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:25.588227 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:25.587999 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:25.588227 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:25.588055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:25.588227 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:25.588122 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:25.588227 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:25.588197 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:25.772309 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:25.772254 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:25.787447 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:25.787422 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:26.936581 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:26.936496 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:27.031945 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.031912 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4r6x"] Nov 01 00:23:27.032103 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.032026 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:27.032150 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:27.032106 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:27.034916 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.034883 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9nfbp"] Nov 01 00:23:27.035079 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.035009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:27.035146 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:27.035098 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:27.035561 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.035523 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2mlf"] Nov 01 00:23:27.035670 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.035642 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:27.035765 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:27.035741 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:27.150080 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.150051 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:27.356903 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.356873 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:27.570717 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.570676 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:27.785987 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.785958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:27.990078 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:27.990047 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:28.200676 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:28.200569 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:28.520469 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:28.520387 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:28.587827 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:28.587785 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:28.588010 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:28.587793 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:28.588010 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:28.587904 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:28.588010 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:28.587785 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:28.588010 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:28.587994 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:28.588158 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:28.588052 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:28.725946 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:28.725918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:29.352850 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:29.352751 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:30.587965 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:30.587770 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:30.588480 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:30.587802 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:30.588480 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:30.588053 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:30.588480 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:30.587866 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:30.588480 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:30.588139 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:30.588480 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:30.588238 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:31.075910 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:31.075872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:31.459609 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:31.459524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:32.011624 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:32.011586 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:32.222784 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:32.222752 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:32.587966 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:32.587923 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:32.588153 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:32.588038 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:32.588153 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:32.588055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:32.588153 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:32.588068 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:32.588258 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:32.588182 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:32.588258 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:32.588224 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:32.592196 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:32.592127 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:32.798676 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:32.798641 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:34.587769 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:34.587732 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:34.588233 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:34.587803 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:34.588233 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:34.587800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:34.588233 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:34.587910 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:34.588233 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:34.588011 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:34.588233 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:34.588074 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:35.504492 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:35.504451 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:35.504686 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:35.504512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:35.504686 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:35.504590 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:35.504686 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:35.504598 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:35.504686 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:35.504652 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret podName:5f953789-9f33-475b-9687-714c6960973c nodeName:}" failed. No retries permitted until 2026-11-01 00:24:07.504637427 +0000 UTC m=+65.381248790 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret") pod "global-pull-secret-syncer-x4r6x" (UID: "5f953789-9f33-475b-9687-714c6960973c") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:35.504686 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:35.504666 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs podName:15c3c93f-5487-4bbe-a237-4528983d8d38 nodeName:}" failed. No retries permitted until 2026-11-01 00:24:07.504659367 +0000 UTC m=+65.381270730 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs") pod "network-metrics-daemon-v2mlf" (UID: "15c3c93f-5487-4bbe-a237-4528983d8d38") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:35.605439 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:35.605402 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:35.605820 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:35.605559 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:23:35.605820 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:35.605575 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:23:35.605820 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:35.605584 2570 projected.go:196] Error preparing data for projected volume kube-api-access-495mv for pod openshift-network-diagnostics/network-check-target-9nfbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:35.605820 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:35.605641 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv podName:bf719590-ce53-4a44-af67-05c687f68258 nodeName:}" failed. No retries permitted until 2026-11-01 00:24:07.605623477 +0000 UTC m=+65.482234859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-495mv" (UniqueName: "kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv") pod "network-check-target-9nfbp" (UID: "bf719590-ce53-4a44-af67-05c687f68258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:35.939956 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:35.939925 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:36.194687 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:36.194602 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:36.424104 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:36.424069 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:36.587371 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:36.587338 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:36.587523 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:36.587376 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:36.587523 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:36.587338 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:36.587523 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:36.587449 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:36.587632 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:36.587543 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:36.587632 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:36.587595 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:36.636816 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:36.636776 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:37.032142 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:37.032065 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f994g_fa5c470e-0673-4c56-a4e8-feb9fa32b67e/dns-node-resolver/0.log" Nov 01 00:23:37.791638 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:37.791596 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:38.009327 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:38.009274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:38.226825 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:38.226738 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:38.231524 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:38.231504 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qvsc4_80ea826e-c81c-4429-9a5d-131a2113fcb3/node-ca/0.log" Nov 01 00:23:38.426767 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:38.426739 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:38.587990 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:38.587954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:38.588170 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:38.587955 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:38.588170 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:38.588087 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:38.588170 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:38.588091 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:38.588279 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:38.588170 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:38.588279 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:38.588232 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:38.631130 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:38.631082 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:39.792156 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:39.792124 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:39.991435 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:39.991401 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:40.199763 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:40.199676 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:40.408158 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:40.408101 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:40.587495 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:40.587459 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:40.587676 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:40.587460 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:40.587676 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:40.587604 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:40.587676 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:40.587463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:40.587676 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:40.587646 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:40.587814 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:40.587715 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:40.613998 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:40.613968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:41.389757 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:41.389729 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:41.791009 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:41.790974 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:41.999992 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:41.999961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:42.205330 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:42.205239 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:42.417646 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:42.417617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:42.588128 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:42.588098 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:42.588308 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:42.588229 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:42.588308 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:42.588272 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:42.588389 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:42.588326 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:42.588424 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:42.588400 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:42.588494 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:42.588466 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:42.628402 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:42.628367 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:43.789241 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:43.789208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:44.000527 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:44.000500 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:44.208714 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:44.208631 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:44.420502 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:44.420466 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:44.587793 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:44.587751 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:44.587977 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:44.587750 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:44.587977 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:44.587876 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:44.587977 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:44.587750 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:44.588134 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:44.587936 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:44.588134 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:44.588058 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:44.635277 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:44.635241 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:44.870195 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:44.870121 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:45.791010 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:45.790978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:45.994509 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:45.994474 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:46.205076 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:46.204998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:46.420713 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:46.420681 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:46.590216 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:46.590187 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:46.590400 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:46.590327 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:46.590400 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:46.590339 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nfbp" podUID="bf719590-ce53-4a44-af67-05c687f68258" Nov 01 00:23:46.590400 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:46.590351 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:46.590499 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:46.590417 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2mlf" podUID="15c3c93f-5487-4bbe-a237-4528983d8d38" Nov 01 00:23:46.590499 ip-10-0-141-93 kubenswrapper[2570]: E1101 00:23:46.590473 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4r6x" podUID="5f953789-9f33-475b-9687-714c6960973c" Nov 01 00:23:46.685047 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:46.685014 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:46.989993 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:46.989916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:47.195809 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.195776 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:47.382953 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.382923 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeReady" Nov 01 00:23:47.383105 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.383058 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 00:23:47.408011 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.407978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:47.499035 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.498996 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pz786"] Nov 01 00:23:47.500630 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.500609 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9rxd6"] Nov 01 00:23:47.500776 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.500756 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:23:47.502420 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.502401 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wt9s4"] Nov 01 00:23:47.502599 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.502520 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.503889 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.503868 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.547780 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.547748 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pz786"] Nov 01 00:23:47.550742 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.550712 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9rxd6"] Nov 01 00:23:47.553582 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.553511 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 00:23:47.553706 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.553590 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 00:23:47.553706 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.553590 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5tc5v\"" Nov 01 00:23:47.553706 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.553636 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 00:23:47.563171 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.563146 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wt9s4"] Nov 01 00:23:47.588843 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.588808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f120a481-1b91-4db7-92c7-eebb3d7e7192-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.588843 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.588846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-metrics-tls\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.589054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.588860 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de6f778b-df02-4af4-9e02-0d7d40321a52-cert\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:23:47.589054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.588886 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkgw\" (UniqueName: \"kubernetes.io/projected/de6f778b-df02-4af4-9e02-0d7d40321a52-kube-api-access-hvkgw\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:23:47.589054 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.589030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-config-volume\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.589147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.589097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.589147 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.589124 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82rg\" (UniqueName: \"kubernetes.io/projected/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-api-access-l82rg\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.589205 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.589160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-tmp-dir\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.589205 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.589196 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f120a481-1b91-4db7-92c7-eebb3d7e7192-crio-socket\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.589273 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.589211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f120a481-1b91-4db7-92c7-eebb3d7e7192-data-volume\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.589344 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.589274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljc9d\" (UniqueName: \"kubernetes.io/projected/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-kube-api-access-ljc9d\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.619479 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.619445 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:47.654133 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.654109 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 00:23:47.654332 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.654307 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 00:23:47.654448 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.654323 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqdjj\"" Nov 01 00:23:47.654448 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.654325 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 00:23:47.654448 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.654356 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 00:23:47.654448 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.654363 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwm8b\"" Nov 01 00:23:47.654448 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.654376 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 00:23:47.657526 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.655082 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 00:23:47.689676 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljc9d\" (UniqueName: \"kubernetes.io/projected/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-kube-api-access-ljc9d\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.689808 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f120a481-1b91-4db7-92c7-eebb3d7e7192-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.689808 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-metrics-tls\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.689808 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689718 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de6f778b-df02-4af4-9e02-0d7d40321a52-cert\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:23:47.689808 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkgw\" (UniqueName: \"kubernetes.io/projected/de6f778b-df02-4af4-9e02-0d7d40321a52-kube-api-access-hvkgw\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:23:47.689808 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689764 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-config-volume\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.689808 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689797 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.690070 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82rg\" (UniqueName: \"kubernetes.io/projected/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-api-access-l82rg\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.690070 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689861 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-tmp-dir\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.690070 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689888 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f120a481-1b91-4db7-92c7-eebb3d7e7192-crio-socket\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.690070 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.689913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f120a481-1b91-4db7-92c7-eebb3d7e7192-data-volume\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.690260 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.690162 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f120a481-1b91-4db7-92c7-eebb3d7e7192-crio-socket\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.690492 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.690472 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f120a481-1b91-4db7-92c7-eebb3d7e7192-data-volume\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.690600 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.690573 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-tmp-dir\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.690600 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.690583 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-config-volume\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.690600 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.690559 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.693379 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.693353 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-metrics-tls\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.693481 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.693389 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f120a481-1b91-4db7-92c7-eebb3d7e7192-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.693481 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.693435 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de6f778b-df02-4af4-9e02-0d7d40321a52-cert\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:23:47.751232 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.751193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljc9d\" (UniqueName: \"kubernetes.io/projected/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-kube-api-access-ljc9d\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:47.751400 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.751270 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82rg\" (UniqueName: \"kubernetes.io/projected/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-api-access-l82rg\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.751475 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.751412 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkgw\" (UniqueName: \"kubernetes.io/projected/de6f778b-df02-4af4-9e02-0d7d40321a52-kube-api-access-hvkgw\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:23:47.811582 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.811496 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:23:47.818928 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.818890 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:23:47.823646 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:47.823624 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:48.252800 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.252728 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pz786"] Nov 01 00:23:48.256674 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.256651 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wt9s4"] Nov 01 00:23:48.257029 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:48.256993 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6f778b_df02_4af4_9e02_0d7d40321a52.slice/crio-af959519697a9d5a96ba08701cf361c58180b14c70b614c45c12cf7b81f938d6 WatchSource:0}: Error finding container af959519697a9d5a96ba08701cf361c58180b14c70b614c45c12cf7b81f938d6: Status 404 returned error can't find the container with id af959519697a9d5a96ba08701cf361c58180b14c70b614c45c12cf7b81f938d6 Nov 01 00:23:48.257344 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.257321 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9rxd6"] Nov 01 00:23:48.259316 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:48.259248 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf120a481_1b91_4db7_92c7_eebb3d7e7192.slice/crio-686938818e898cee1468a8034c45c472a9829dce7b8043b88413fb089af4cd03 WatchSource:0}: Error finding container 686938818e898cee1468a8034c45c472a9829dce7b8043b88413fb089af4cd03: Status 404 returned error can't find the container with id 686938818e898cee1468a8034c45c472a9829dce7b8043b88413fb089af4cd03 Nov 01 00:23:48.260171 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:48.260150 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc154cf87_34c2_411a_bf51_a67f2cd9e1d2.slice/crio-207043ddccbe780b1664dc5d87a9f9f9d9cd2218fd72997089fee24aecf61db7 WatchSource:0}: Error finding container 207043ddccbe780b1664dc5d87a9f9f9d9cd2218fd72997089fee24aecf61db7: Status 404 returned error can't find the container with id 207043ddccbe780b1664dc5d87a9f9f9d9cd2218fd72997089fee24aecf61db7 Nov 01 00:23:48.587093 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.587052 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:23:48.587324 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.587063 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:23:48.587324 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.587193 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:23:48.639661 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.639627 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 00:23:48.639823 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.639688 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 00:23:48.640175 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.640157 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 00:23:48.640243 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.640165 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 00:23:48.640243 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.640189 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t74k4\"" Nov 01 00:23:48.640243 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.640217 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7ckpz\"" Nov 01 00:23:48.640356 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.640306 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rjfbh\"" Nov 01 00:23:48.823171 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.823129 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt9s4" event={"ID":"c154cf87-34c2-411a-bf51-a67f2cd9e1d2","Type":"ContainerStarted","Data":"207043ddccbe780b1664dc5d87a9f9f9d9cd2218fd72997089fee24aecf61db7"} Nov 01 00:23:48.824614 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.824592 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rxd6" event={"ID":"f120a481-1b91-4db7-92c7-eebb3d7e7192","Type":"ContainerStarted","Data":"657791ef27535145f3dc397ce141cfb88be06d9cfd4cbc88f9c8b942a3acadb8"} Nov 01 00:23:48.824745 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.824643 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rxd6" event={"ID":"f120a481-1b91-4db7-92c7-eebb3d7e7192","Type":"ContainerStarted","Data":"686938818e898cee1468a8034c45c472a9829dce7b8043b88413fb089af4cd03"} Nov 01 00:23:48.825740 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:48.825721 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pz786" event={"ID":"de6f778b-df02-4af4-9e02-0d7d40321a52","Type":"ContainerStarted","Data":"af959519697a9d5a96ba08701cf361c58180b14c70b614c45c12cf7b81f938d6"} Nov 01 00:23:49.114518 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:49.114481 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:49.323448 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:49.323412 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:49.831204 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:49.831161 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rxd6" event={"ID":"f120a481-1b91-4db7-92c7-eebb3d7e7192","Type":"ContainerStarted","Data":"63e506b435ad4911058b985343f8c8cd1aae4534090b860ec1c491b83e20d71c"} Nov 01 00:23:50.150785 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.150702 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:50.361198 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.361155 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:50.506302 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.506244 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz"] Nov 01 00:23:50.508873 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.508853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.561656 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.561598 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 00:23:50.561656 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.561598 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 00:23:50.595440 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.595344 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.612751 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.612816 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-kube-api-access-8x5g6\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.612879 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.612905 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.613353 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.613535 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.613652 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dqtrs\"" Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.613705 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 00:23:50.614043 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.613737 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz"] Nov 01 00:23:50.714562 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.714533 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.714670 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.714574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-kube-api-access-8x5g6\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.714670 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.714617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.715510 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.715483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.715785 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.715761 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.719175 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.719147 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.719887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.719369 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.776901 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.776859 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-kube-api-access-8x5g6\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.808020 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.807972 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:50.836873 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.836830 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt9s4" event={"ID":"c154cf87-34c2-411a-bf51-a67f2cd9e1d2","Type":"ContainerStarted","Data":"e4530efc472461e06ae0cc29320aaa2a916026ebda08cc561ca4f86a782fa6da"} Nov 01 00:23:50.837062 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.836893 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt9s4" event={"ID":"c154cf87-34c2-411a-bf51-a67f2cd9e1d2","Type":"ContainerStarted","Data":"133cf463656be6b40286adebef9d74cc9efabd91a4a27554c73193d8204616ba"} Nov 01 00:23:50.837062 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.836991 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wt9s4" Nov 01 00:23:50.886734 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.886704 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:23:50.966931 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:50.966861 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-wt9s4" podStartSLOduration=1.777581488 podStartE2EDuration="3.966841948s" totalImagesPullingTime="2.18926046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:50.966525897 +0000 UTC m=+48.843137283" watchObservedRunningTime="2026-11-01 00:23:50.966841948 +0000 UTC m=+48.843453334" Nov 01 00:23:51.273729 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:51.273701 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz"] Nov 01 00:23:51.276813 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:51.276777 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ea6ad3_b91d_4c5c_b0ac_7974d9dab56b.slice/crio-36c62bd8c0d4ca31b0e2e148fb96cd70e52355b583aec8610548c1d8c6a4ef94 WatchSource:0}: Error finding container 36c62bd8c0d4ca31b0e2e148fb96cd70e52355b583aec8610548c1d8c6a4ef94: Status 404 returned error can't find the container with id 36c62bd8c0d4ca31b0e2e148fb96cd70e52355b583aec8610548c1d8c6a4ef94 Nov 01 00:23:51.841407 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:51.841377 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rxd6" event={"ID":"f120a481-1b91-4db7-92c7-eebb3d7e7192","Type":"ContainerStarted","Data":"21bf824fac7225cee5027f8a4d28d8f41587ab4e2dd3bb077c9063ec864a455d"} Nov 01 00:23:51.842706 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:51.842644 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" event={"ID":"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b","Type":"ContainerStarted","Data":"36c62bd8c0d4ca31b0e2e148fb96cd70e52355b583aec8610548c1d8c6a4ef94"} Nov 01 00:23:51.843990 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:51.843964 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pz786" event={"ID":"de6f778b-df02-4af4-9e02-0d7d40321a52","Type":"ContainerStarted","Data":"3d24e85e0d84f0dae761bed1738b7ea2aaf97da764c9e18c0b548c415a0042a0"} Nov 01 00:23:51.959640 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:51.959592 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9rxd6" podStartSLOduration=1.7726595939999998 podStartE2EDuration="4.959577173s" totalImagesPullingTime="3.186917579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:51.958791767 +0000 UTC m=+49.835403153" watchObservedRunningTime="2026-11-01 00:23:51.959577173 +0000 UTC m=+49.836188536" Nov 01 00:23:52.073565 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.073508 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pz786" podStartSLOduration=1.943569952 podStartE2EDuration="5.073492892s" totalImagesPullingTime="3.12992294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:52.073213278 +0000 UTC m=+49.949824663" watchObservedRunningTime="2026-11-01 00:23:52.073492892 +0000 UTC m=+49.950104277" Nov 01 00:23:52.137617 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.137546 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5665b49b4d-hvmbv"] Nov 01 00:23:52.139114 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.139097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.191666 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.191629 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 00:23:52.191666 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.191634 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 00:23:52.191666 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.191645 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 00:23:52.191955 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.191779 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 00:23:52.191955 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.191801 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 00:23:52.191955 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.191838 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d652d\"" Nov 01 00:23:52.191955 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.191856 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 00:23:52.191955 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.191942 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 00:23:52.200183 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.200132 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5665b49b4d-hvmbv"] Nov 01 00:23:52.227268 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.227228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.227480 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.227279 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9g6\" (UniqueName: \"kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.227480 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.227335 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.227480 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.227388 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.227480 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.227427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.227651 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.227492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.227651 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.227548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.328719 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.328685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.328719 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.328727 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.328985 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.328752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.328985 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.328781 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9g6\" (UniqueName: \"kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.328985 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.328801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.328985 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.328815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.328985 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.328884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.329638 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.329561 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.329638 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.329655 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.329638 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.329655 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.329985 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.329751 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.331425 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.331401 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.331601 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.331584 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.389769 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.389689 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9g6\" (UniqueName: \"kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.448845 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.448810 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:23:52.833758 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.833725 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5665b49b4d-hvmbv"] Nov 01 00:23:52.837345 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:52.837309 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7918db7e_5d6b_47bd_bd67_32811f8e40f7.slice/crio-ce2a983f542730942e817f2aab8cc51ef2c45b446a1dbd19ab7a3a7ab511c7c8 WatchSource:0}: Error finding container ce2a983f542730942e817f2aab8cc51ef2c45b446a1dbd19ab7a3a7ab511c7c8: Status 404 returned error can't find the container with id ce2a983f542730942e817f2aab8cc51ef2c45b446a1dbd19ab7a3a7ab511c7c8 Nov 01 00:23:52.847986 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:52.847952 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5665b49b4d-hvmbv" event={"ID":"7918db7e-5d6b-47bd-bd67-32811f8e40f7","Type":"ContainerStarted","Data":"ce2a983f542730942e817f2aab8cc51ef2c45b446a1dbd19ab7a3a7ab511c7c8"} Nov 01 00:23:53.854366 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:53.854324 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" event={"ID":"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b","Type":"ContainerStarted","Data":"68062cc53349c4186b231574157d20a4addd7911252e82d2b4ba96ae8171c77c"} Nov 01 00:23:53.854770 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:53.854399 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" event={"ID":"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b","Type":"ContainerStarted","Data":"4c618390cc94659481048090d32e2d7ade133252e4de85b9dfd84d2d84dcf813"} Nov 01 00:23:53.972400 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:53.972339 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" podStartSLOduration=2.183492984 podStartE2EDuration="3.972324454s" totalImagesPullingTime="1.78883147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:53.971019041 +0000 UTC m=+51.847630428" watchObservedRunningTime="2026-11-01 00:23:53.972324454 +0000 UTC m=+51.848935869" Nov 01 00:23:55.808494 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:55.808467 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:55.869759 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:55.869717 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5665b49b4d-hvmbv" event={"ID":"7918db7e-5d6b-47bd-bd67-32811f8e40f7","Type":"ContainerStarted","Data":"549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a"} Nov 01 00:23:56.013621 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:56.013574 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5665b49b4d-hvmbv" podStartSLOduration=1.067875619 podStartE2EDuration="4.01356066s" totalImagesPullingTime="2.945685041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:56.012857102 +0000 UTC m=+53.889468504" watchObservedRunningTime="2026-11-01 00:23:56.01356066 +0000 UTC m=+53.890172045" Nov 01 00:23:56.463979 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:56.463939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:56.755232 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:56.755157 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:56.963467 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:56.963430 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:57.126417 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.126379 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn"] Nov 01 00:23:57.129462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.129445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.174132 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.174098 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:57.231536 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.231502 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 00:23:57.231536 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.231501 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d9w4t\"" Nov 01 00:23:57.231739 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.231554 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 00:23:57.234100 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.234069 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn"] Nov 01 00:23:57.273977 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.273935 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.273977 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.273976 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6qp\" (UniqueName: \"kubernetes.io/projected/6df8f53e-1025-4a88-8945-4055732800c6-kube-api-access-hf6qp\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.274211 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.274089 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6df8f53e-1025-4a88-8945-4055732800c6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.274211 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.274141 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.282545 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.282519 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf"] Nov 01 00:23:57.285844 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.285820 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5d695"] Nov 01 00:23:57.285962 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.285948 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.288861 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.288842 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.338181 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.338143 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vqqjw\"" Nov 01 00:23:57.338181 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.338161 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 00:23:57.338181 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.338142 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 00:23:57.340699 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.340681 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 00:23:57.340795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.340782 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 00:23:57.340907 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.340896 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vngfn\"" Nov 01 00:23:57.347422 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.347397 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf"] Nov 01 00:23:57.375452 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.375656 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6df8f53e-1025-4a88-8945-4055732800c6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.375656 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.375656 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6qp\" (UniqueName: \"kubernetes.io/projected/6df8f53e-1025-4a88-8945-4055732800c6-kube-api-access-hf6qp\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.375656 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375602 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.375656 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stflp\" (UniqueName: \"kubernetes.io/projected/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-api-access-stflp\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.375913 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375686 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.375913 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375703 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.375913 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.375913 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.375819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.376308 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.376260 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6df8f53e-1025-4a88-8945-4055732800c6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.378325 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.378255 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.378325 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.378303 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.387823 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.387792 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:57.423205 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.423173 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 00:23:57.423400 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.423219 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 00:23:57.436849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.436812 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6qp\" (UniqueName: \"kubernetes.io/projected/6df8f53e-1025-4a88-8945-4055732800c6-kube-api-access-hf6qp\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.444466 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.444429 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:23:57.477149 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.477328 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-accelerators-collector-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.477328 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.477328 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477217 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-tls\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.477328 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stflp\" (UniqueName: \"kubernetes.io/projected/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-api-access-stflp\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.477530 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477501 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgkh5\" (UniqueName: \"kubernetes.io/projected/afca2ec8-21f4-4bb6-bc31-009191b47dca-kube-api-access-pgkh5\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.477577 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-root\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.477614 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-metrics-client-ca\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.477707 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477687 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-sys\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.477803 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477780 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.477859 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477838 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.477910 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-wtmp\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.477958 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.477958 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-textfile\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.478049 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477966 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.478097 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.477874 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.478163 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.478136 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.478586 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.478565 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.479996 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.479970 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.480190 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.480172 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.538687 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.538652 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stflp\" (UniqueName: \"kubernetes.io/projected/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-api-access-stflp\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.578625 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-root\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578625 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-metrics-client-ca\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578804 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-sys\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578804 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578708 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-sys\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578804 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578704 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-root\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578804 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578766 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-wtmp\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578804 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-textfile\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578970 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578817 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578970 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578856 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-accelerators-collector-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578970 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578878 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-wtmp\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578970 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-tls\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.578970 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.578958 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkh5\" (UniqueName: \"kubernetes.io/projected/afca2ec8-21f4-4bb6-bc31-009191b47dca-kube-api-access-pgkh5\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.579217 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.579198 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-textfile\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.579337 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.579316 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-metrics-client-ca\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.579424 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.579408 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-accelerators-collector-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.581351 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.581333 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.581420 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.581406 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-tls\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.598791 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.598762 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:57.602683 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.602664 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:23:57.641412 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.641332 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkh5\" (UniqueName: \"kubernetes.io/projected/afca2ec8-21f4-4bb6-bc31-009191b47dca-kube-api-access-pgkh5\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.789033 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.789006 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:23:57.805177 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.805133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:57.868610 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.868581 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn"] Nov 01 00:23:57.871073 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:57.871046 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df8f53e_1025_4a88_8945_4055732800c6.slice/crio-0ddc0719a785852a8f3464f1052c47bf5d7fa10df9fa06605370c2d67a77c057 WatchSource:0}: Error finding container 0ddc0719a785852a8f3464f1052c47bf5d7fa10df9fa06605370c2d67a77c057: Status 404 returned error can't find the container with id 0ddc0719a785852a8f3464f1052c47bf5d7fa10df9fa06605370c2d67a77c057 Nov 01 00:23:57.876173 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.876149 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" event={"ID":"6df8f53e-1025-4a88-8945-4055732800c6","Type":"ContainerStarted","Data":"0ddc0719a785852a8f3464f1052c47bf5d7fa10df9fa06605370c2d67a77c057"} Nov 01 00:23:57.908432 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.908402 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:23:57.920689 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:57.920653 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafca2ec8_21f4_4bb6_bc31_009191b47dca.slice/crio-68791c348fd78f0f47eea65b047ee982cc236034921c439185b168cbbc024f33 WatchSource:0}: Error finding container 68791c348fd78f0f47eea65b047ee982cc236034921c439185b168cbbc024f33: Status 404 returned error can't find the container with id 68791c348fd78f0f47eea65b047ee982cc236034921c439185b168cbbc024f33 Nov 01 00:23:57.972975 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:57.972875 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf"] Nov 01 00:23:57.975270 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:23:57.975247 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5b2a72_a248_4a35_b6f3_e13c5f7b015b.slice/crio-48eadde52d4b4d9834ec26c6765bab3ac53bad6d8ebee4aeb991c8ec2a88fd0a WatchSource:0}: Error finding container 48eadde52d4b4d9834ec26c6765bab3ac53bad6d8ebee4aeb991c8ec2a88fd0a: Status 404 returned error can't find the container with id 48eadde52d4b4d9834ec26c6765bab3ac53bad6d8ebee4aeb991c8ec2a88fd0a Nov 01 00:23:58.579738 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:58.579708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:58.787126 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:58.787090 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:58.881161 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:58.881080 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" event={"ID":"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b","Type":"ContainerStarted","Data":"48eadde52d4b4d9834ec26c6765bab3ac53bad6d8ebee4aeb991c8ec2a88fd0a"} Nov 01 00:23:58.883573 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:58.883547 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" event={"ID":"6df8f53e-1025-4a88-8945-4055732800c6","Type":"ContainerStarted","Data":"7b20f2978145968867c7bad5210f0b86832ade219d94f75547e7ff65f9345d4f"} Nov 01 00:23:58.883727 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:58.883602 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" event={"ID":"6df8f53e-1025-4a88-8945-4055732800c6","Type":"ContainerStarted","Data":"023c5ef70bf4a214b4e11c2337e6f9ca2737e55167a746ce485b293e18d529a4"} Nov 01 00:23:58.884903 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:58.884882 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5d695" event={"ID":"afca2ec8-21f4-4bb6-bc31-009191b47dca","Type":"ContainerStarted","Data":"68791c348fd78f0f47eea65b047ee982cc236034921c439185b168cbbc024f33"} Nov 01 00:23:59.471462 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.471428 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:23:59.475117 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.475090 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.528352 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.527981 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 00:23:59.528352 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.528025 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 00:23:59.528352 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.528101 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 00:23:59.528352 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.528133 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8rsns\"" Nov 01 00:23:59.528352 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.528209 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 00:23:59.528352 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.528307 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 00:23:59.528676 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.528385 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 00:23:59.533916 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.533888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:23:59.579427 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.579396 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:23:59.597027 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.596932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597027 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.596987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597315 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2q7j\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597315 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597315 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597453 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597453 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597453 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597453 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597432 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597483 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597514 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.597648 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.597542 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.628642 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.628577 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 00:23:59.674216 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.674150 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 00:23:59.674216 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.674175 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 00:23:59.698572 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698540 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698572 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698577 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.698795 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.699275 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698811 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.699275 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2q7j\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.699275 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.698873 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.699459 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.699393 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.700170 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.699840 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.700170 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.699881 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.703137 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.702691 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.703137 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.702927 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.703870 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.703391 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.703870 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.703589 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.703870 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.703840 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.704016 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.703956 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.704143 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.704120 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.704861 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.704832 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.705594 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.705573 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.763269 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.763241 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2q7j\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.790413 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.790381 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-546d87964-scgxb"] Nov 01 00:23:59.795539 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.795511 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.807496 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.807471 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:23:59.848161 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.848111 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 00:23:59.848236 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.848161 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 00:23:59.848236 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.848209 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 00:23:59.848236 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.848220 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 00:23:59.848236 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.848215 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 00:23:59.848449 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.848298 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wfz2h\"" Nov 01 00:23:59.848449 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.848371 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-75fpm3fqiecoh\"" Nov 01 00:23:59.854250 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.854207 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-546d87964-scgxb"] Nov 01 00:23:59.895089 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.895056 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" event={"ID":"6df8f53e-1025-4a88-8945-4055732800c6","Type":"ContainerStarted","Data":"dc301072d3d65cf5c892d0f0362ed30316a18e365fbd6fbcf5c498cd7bbda30d"} Nov 01 00:23:59.898560 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.898530 2570 generic.go:356] "Generic (PLEG): container finished" podID="afca2ec8-21f4-4bb6-bc31-009191b47dca" containerID="fb07ae7f06ab5e064a6056ee9606e94fbafa312130dbe9836cdf0407c14e2cfc" exitCode=0 Nov 01 00:23:59.898671 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.898603 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5d695" event={"ID":"afca2ec8-21f4-4bb6-bc31-009191b47dca","Type":"ContainerDied","Data":"fb07ae7f06ab5e064a6056ee9606e94fbafa312130dbe9836cdf0407c14e2cfc"} Nov 01 00:23:59.900772 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.900750 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9deeffc-4dcb-489c-bbbd-f472272d3c13-tmp\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.900816 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.900794 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9deeffc-4dcb-489c-bbbd-f472272d3c13-metrics-client-ca\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.900919 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.900902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.900952 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.900942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.900994 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.900983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjr4\" (UniqueName: \"kubernetes.io/projected/d9deeffc-4dcb-489c-bbbd-f472272d3c13-kube-api-access-5pjr4\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.901024 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.901012 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-grpc-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.901065 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.901053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.901095 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.901085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:23:59.901126 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:23:59.901101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002349 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002315 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjr4\" (UniqueName: \"kubernetes.io/projected/d9deeffc-4dcb-489c-bbbd-f472272d3c13-kube-api-access-5pjr4\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002454 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-grpc-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002521 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002494 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002576 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002558 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002625 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002673 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9deeffc-4dcb-489c-bbbd-f472272d3c13-tmp\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002718 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9deeffc-4dcb-489c-bbbd-f472272d3c13-metrics-client-ca\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002773 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002740 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.002835 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.002818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.004753 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.004397 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9deeffc-4dcb-489c-bbbd-f472272d3c13-metrics-client-ca\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.005866 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.005836 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9deeffc-4dcb-489c-bbbd-f472272d3c13-tmp\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.006719 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.006694 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.007162 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.007143 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.008624 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.008594 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.009098 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.009074 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.010801 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.010777 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.010885 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.010799 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-grpc-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.036914 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.036859 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" podStartSLOduration=1.311682888 podStartE2EDuration="3.036843374s" totalImagesPullingTime="1.725160486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:00.0354246 +0000 UTC m=+57.912035985" watchObservedRunningTime="2026-11-01 00:24:00.036843374 +0000 UTC m=+57.913454758" Nov 01 00:24:00.066874 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.066825 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjr4\" (UniqueName: \"kubernetes.io/projected/d9deeffc-4dcb-489c-bbbd-f472272d3c13-kube-api-access-5pjr4\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.121063 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.121023 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:00.267105 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.267081 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:24:00.269756 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:00.269729 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144d7167_16c0_4f04_a304_ebc22c47283e.slice/crio-401f1de8b558af3b8ac80748f24fc3ea813e9e99a1fbf78be70e6a08dac1624f WatchSource:0}: Error finding container 401f1de8b558af3b8ac80748f24fc3ea813e9e99a1fbf78be70e6a08dac1624f: Status 404 returned error can't find the container with id 401f1de8b558af3b8ac80748f24fc3ea813e9e99a1fbf78be70e6a08dac1624f Nov 01 00:24:00.501001 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.500709 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-546d87964-scgxb"] Nov 01 00:24:00.503237 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:00.503210 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9deeffc_4dcb_489c_bbbd_f472272d3c13.slice/crio-476c72a3effe87e598627cb59ab7085f34ffd9d23ae23bd3c0e34c6b94cd451b WatchSource:0}: Error finding container 476c72a3effe87e598627cb59ab7085f34ffd9d23ae23bd3c0e34c6b94cd451b: Status 404 returned error can't find the container with id 476c72a3effe87e598627cb59ab7085f34ffd9d23ae23bd3c0e34c6b94cd451b Nov 01 00:24:00.846358 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.846327 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:00.903839 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.903798 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5d695" event={"ID":"afca2ec8-21f4-4bb6-bc31-009191b47dca","Type":"ContainerStarted","Data":"394b3c6df8bff77953ccb65c6c496012a1afcd35be40662182271ee54e7b29c9"} Nov 01 00:24:00.904033 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.903867 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5d695" event={"ID":"afca2ec8-21f4-4bb6-bc31-009191b47dca","Type":"ContainerStarted","Data":"969879bba225a804428bca9b6d3a67f49db6269526b55f1eabd77ee6cbaf02a6"} Nov 01 00:24:00.905877 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.905852 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" event={"ID":"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b","Type":"ContainerStarted","Data":"7c63867aad1ef1791536d705e400f9e8e16e09efd503f762d728469b98e1a4d8"} Nov 01 00:24:00.906023 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.905897 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" event={"ID":"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b","Type":"ContainerStarted","Data":"b6d3a054cd91994336283d29cb3536a3d048e3476a74e4769b330bfb7ba1ec1a"} Nov 01 00:24:00.906023 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.905921 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" event={"ID":"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b","Type":"ContainerStarted","Data":"f5d35fbef904cc72a0007d0b31fcf777180638e2d48f53ef3b976ae51725595b"} Nov 01 00:24:00.907624 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.907593 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" event={"ID":"d9deeffc-4dcb-489c-bbbd-f472272d3c13","Type":"ContainerStarted","Data":"476c72a3effe87e598627cb59ab7085f34ffd9d23ae23bd3c0e34c6b94cd451b"} Nov 01 00:24:00.909888 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:00.909850 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerStarted","Data":"401f1de8b558af3b8ac80748f24fc3ea813e9e99a1fbf78be70e6a08dac1624f"} Nov 01 00:24:01.085723 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.085670 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" podStartSLOduration=2.21883349 podStartE2EDuration="4.085654874s" totalImagesPullingTime="1.866821384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:01.085120371 +0000 UTC m=+58.961731753" watchObservedRunningTime="2026-11-01 00:24:01.085654874 +0000 UTC m=+58.962266258" Nov 01 00:24:01.205015 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.204924 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5d695" podStartSLOduration=2.636045729 podStartE2EDuration="4.20490829s" totalImagesPullingTime="1.473160345s" totalInitContainerRuntime="95.702216ms" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:01.203846669 +0000 UTC m=+59.080458053" watchObservedRunningTime="2026-11-01 00:24:01.20490829 +0000 UTC m=+59.081519674" Nov 01 00:24:01.273162 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.273125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:01.537923 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.537879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:01.747224 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.747192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:01.906805 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.906714 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d85jb"] Nov 01 00:24:01.910051 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.910027 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:01.920903 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.920872 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwwm\" (UniqueName: \"kubernetes.io/projected/34995d02-7971-4439-9d1c-c1083cc72be9-kube-api-access-crwwm\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:01.921098 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.921078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34995d02-7971-4439-9d1c-c1083cc72be9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:01.951766 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.951739 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:01.962703 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.962678 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-b2d4t\"" Nov 01 00:24:01.962703 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.962678 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 00:24:01.968146 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:01.968114 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d85jb"] Nov 01 00:24:02.022267 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.022234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwwm\" (UniqueName: \"kubernetes.io/projected/34995d02-7971-4439-9d1c-c1083cc72be9-kube-api-access-crwwm\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:02.022455 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.022348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34995d02-7971-4439-9d1c-c1083cc72be9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:02.027104 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.027072 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34995d02-7971-4439-9d1c-c1083cc72be9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:02.060857 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.060825 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5db84b6cc6-rp9xt"] Nov 01 00:24:02.064139 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.064109 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.083732 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.083699 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwwm\" (UniqueName: \"kubernetes.io/projected/34995d02-7971-4439-9d1c-c1083cc72be9-kube-api-access-crwwm\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:02.116897 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.116861 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 00:24:02.117075 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.117061 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xl8bk\"" Nov 01 00:24:02.117140 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.117081 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-936to0tetgr6n\"" Nov 01 00:24:02.117140 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.117093 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 00:24:02.117140 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.117116 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 00:24:02.117140 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.117133 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 00:24:02.123433 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.123404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-client-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.123600 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.123451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.123600 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.123483 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5db84b6cc6-rp9xt"] Nov 01 00:24:02.123600 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.123495 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ktk\" (UniqueName: \"kubernetes.io/projected/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-kube-api-access-t8ktk\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.123600 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.123523 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-client-certs\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.123600 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.123548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-audit-log\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.123838 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.123634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-tls\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.123838 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.123703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-metrics-server-audit-profiles\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.160608 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.160525 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:02.224482 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.224448 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:02.224663 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.224620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ktk\" (UniqueName: \"kubernetes.io/projected/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-kube-api-access-t8ktk\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.224727 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.224677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-client-certs\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.224727 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.224708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-audit-log\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.224831 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.224754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-tls\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.224831 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.224800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-metrics-server-audit-profiles\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.224929 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.224863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-client-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.224929 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.224918 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.225213 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.225188 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-audit-log\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.225679 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.225633 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.226167 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.226142 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-metrics-server-audit-profiles\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.227823 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.227800 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-client-certs\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.227823 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.227810 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-client-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.235720 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.235694 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-tls\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.289937 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.287432 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ktk\" (UniqueName: \"kubernetes.io/projected/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-kube-api-access-t8ktk\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.396571 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.396533 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:02.449353 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.449259 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:02.450075 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.449567 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:02.456160 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.456007 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:02.600581 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.600547 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d85jb"] Nov 01 00:24:02.641483 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:02.641448 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34995d02_7971_4439_9d1c_c1083cc72be9.slice/crio-80471ee0f664100ace069d095e8fe35c6294918d1ed498f111a85d172307fe30 WatchSource:0}: Error finding container 80471ee0f664100ace069d095e8fe35c6294918d1ed498f111a85d172307fe30: Status 404 returned error can't find the container with id 80471ee0f664100ace069d095e8fe35c6294918d1ed498f111a85d172307fe30 Nov 01 00:24:02.778365 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.778259 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5db84b6cc6-rp9xt"] Nov 01 00:24:02.782508 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:02.782473 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf066bd_0ff4_471d_ba92_d8cd3d6820fe.slice/crio-8a277d0a81f7bfedcc7b14be37afbb5d413453332fba41bc021b3415c3cf6ac3 WatchSource:0}: Error finding container 8a277d0a81f7bfedcc7b14be37afbb5d413453332fba41bc021b3415c3cf6ac3: Status 404 returned error can't find the container with id 8a277d0a81f7bfedcc7b14be37afbb5d413453332fba41bc021b3415c3cf6ac3 Nov 01 00:24:02.919006 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.918982 2570 generic.go:356] "Generic (PLEG): container finished" podID="144d7167-16c0-4f04-a304-ebc22c47283e" containerID="fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169" exitCode=0 Nov 01 00:24:02.919116 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.919068 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerDied","Data":"fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169"} Nov 01 00:24:02.920362 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.920314 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" event={"ID":"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe","Type":"ContainerStarted","Data":"8a277d0a81f7bfedcc7b14be37afbb5d413453332fba41bc021b3415c3cf6ac3"} Nov 01 00:24:02.921492 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.921470 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" event={"ID":"34995d02-7971-4439-9d1c-c1083cc72be9","Type":"ContainerStarted","Data":"80471ee0f664100ace069d095e8fe35c6294918d1ed498f111a85d172307fe30"} Nov 01 00:24:02.925666 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:02.925635 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:03.320941 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.320851 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:03.525086 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.525056 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:03.630673 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.630587 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-585bf7664f-h64rc"] Nov 01 00:24:03.636098 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.636068 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.693830 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.693795 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585bf7664f-h64rc"] Nov 01 00:24:03.736499 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.736462 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.736660 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.736509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.736660 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.736539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.736766 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.736671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxjh\" (UniqueName: \"kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.736766 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.736731 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.736897 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.736875 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.736950 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.736937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.737042 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.737015 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.741870 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.741844 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 00:24:03.838209 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.838174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.838426 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.838229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.838426 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.838380 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.838426 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.838416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.838583 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.838445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.838583 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.838469 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.838583 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.838504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxjh\" (UniqueName: \"kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.838583 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.838569 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.839057 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.839023 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.839275 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.839225 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.839748 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.839715 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.839842 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.839749 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.840221 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.840193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.841917 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.841893 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.842065 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.842006 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.900761 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.900697 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxjh\" (UniqueName: \"kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:03.928333 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.928273 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" event={"ID":"d9deeffc-4dcb-489c-bbbd-f472272d3c13","Type":"ContainerStarted","Data":"b82937fbd35bd0c66904e8c102d31583e0074c0e1596ef6f5dea8a6e07e73fec"} Nov 01 00:24:03.928519 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.928375 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" event={"ID":"d9deeffc-4dcb-489c-bbbd-f472272d3c13","Type":"ContainerStarted","Data":"b787741c7133d3bb3e3669722a3970a2e56e140eebb67e812ef5deee7cd97237"} Nov 01 00:24:03.928519 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.928410 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" event={"ID":"d9deeffc-4dcb-489c-bbbd-f472272d3c13","Type":"ContainerStarted","Data":"c8650dd0cf7faa5889ac9169552ce20d272733a7956183dcb82193cb40b480c0"} Nov 01 00:24:03.949407 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:03.949140 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:04.329400 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.329351 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585bf7664f-h64rc"] Nov 01 00:24:04.359861 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:04.359824 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e31e6c_2296_4e75_9eed_be8579a717b9.slice/crio-ba4f0543a9d2da11d38ed09e342bf56da79f0291f08c95373d2362cb63c40865 WatchSource:0}: Error finding container ba4f0543a9d2da11d38ed09e342bf56da79f0291f08c95373d2362cb63c40865: Status 404 returned error can't find the container with id ba4f0543a9d2da11d38ed09e342bf56da79f0291f08c95373d2362cb63c40865 Nov 01 00:24:04.934670 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.934591 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" event={"ID":"34995d02-7971-4439-9d1c-c1083cc72be9","Type":"ContainerStarted","Data":"7d537c327d177e75a999a2b93b48ba8d853e67a6f9d5be7955c9d082b63c1c35"} Nov 01 00:24:04.935112 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.935011 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:04.937108 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.937058 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585bf7664f-h64rc" event={"ID":"33e31e6c-2296-4e75-9eed-be8579a717b9","Type":"ContainerStarted","Data":"b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077"} Nov 01 00:24:04.937267 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.937113 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585bf7664f-h64rc" event={"ID":"33e31e6c-2296-4e75-9eed-be8579a717b9","Type":"ContainerStarted","Data":"ba4f0543a9d2da11d38ed09e342bf56da79f0291f08c95373d2362cb63c40865"} Nov 01 00:24:04.941092 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.941070 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:04.942004 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.941980 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" event={"ID":"d9deeffc-4dcb-489c-bbbd-f472272d3c13","Type":"ContainerStarted","Data":"2c7046057c779f609ebc62f57ed87cc98a1185dd79e4d1e1991b209dfe125dc4"} Nov 01 00:24:04.942118 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.942044 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" event={"ID":"d9deeffc-4dcb-489c-bbbd-f472272d3c13","Type":"ContainerStarted","Data":"e5ea729bb705671688c3100e7c84658e27fbe3237e510a7f0e046626aedc813c"} Nov 01 00:24:04.942118 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.942085 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" event={"ID":"d9deeffc-4dcb-489c-bbbd-f472272d3c13","Type":"ContainerStarted","Data":"e5ebdf0d00a4c4b95b18271e7aa66d569d42c5d2504cd299118f588543277acf"} Nov 01 00:24:04.942251 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:04.942218 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:05.053557 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.053502 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" podStartSLOduration=2.047892253 podStartE2EDuration="4.053482572s" totalImagesPullingTime="2.005590319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:05.0521583 +0000 UTC m=+62.928769685" watchObservedRunningTime="2026-11-01 00:24:05.053482572 +0000 UTC m=+62.930093957" Nov 01 00:24:05.322102 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.322038 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-585bf7664f-h64rc" podStartSLOduration=2.322020886 podStartE2EDuration="2.322020886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:05.320831276 +0000 UTC m=+63.197442660" watchObservedRunningTime="2026-11-01 00:24:05.322020886 +0000 UTC m=+63.198632272" Nov 01 00:24:05.447811 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.447757 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" podStartSLOduration=2.6382169920000003 podStartE2EDuration="6.447741522s" totalImagesPullingTime="3.80952453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:05.446474809 +0000 UTC m=+63.323086222" watchObservedRunningTime="2026-11-01 00:24:05.447741522 +0000 UTC m=+63.324352907" Nov 01 00:24:05.949271 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.949182 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerStarted","Data":"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4"} Nov 01 00:24:05.949740 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.949320 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerStarted","Data":"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d"} Nov 01 00:24:05.949740 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.949367 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerStarted","Data":"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72"} Nov 01 00:24:05.949740 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.949408 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerStarted","Data":"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b"} Nov 01 00:24:05.949740 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.949447 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerStarted","Data":"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1"} Nov 01 00:24:05.949740 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.949486 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerStarted","Data":"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010"} Nov 01 00:24:05.951071 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:05.951047 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" event={"ID":"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe","Type":"ContainerStarted","Data":"5ce5c737789d4df56419737ea268d49d58c97b81559ab59ce7de927d1c5b344e"} Nov 01 00:24:06.079135 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.079075 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.901305399 podStartE2EDuration="7.079061338s" totalImagesPullingTime="4.160577081s" totalInitContainerRuntime="17.178858ms" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:06.078539171 +0000 UTC m=+63.955150568" watchObservedRunningTime="2026-11-01 00:24:06.079061338 +0000 UTC m=+63.955672723" Nov 01 00:24:06.200935 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.200779 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" podStartSLOduration=1.898893599 podStartE2EDuration="4.200759735s" totalImagesPullingTime="2.301866136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:06.199235013 +0000 UTC m=+64.075846408" watchObservedRunningTime="2026-11-01 00:24:06.200759735 +0000 UTC m=+64.077371121" Nov 01 00:24:06.244162 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.244124 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:24:06.248755 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.248729 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261638 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261772 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261679 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261772 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261772 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261772 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261766 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261986 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261856 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261986 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261885 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261986 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261986 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.261986 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261959 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262219 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.261988 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262219 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262024 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g85t\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262219 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262132 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262219 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262219 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262193 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262219 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262213 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262537 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262352 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262537 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262537 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.262537 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.262438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.304764 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.304723 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 00:24:06.304931 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.304900 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 00:24:06.305272 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305200 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 00:24:06.305272 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305207 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 00:24:06.305272 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305225 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 00:24:06.305272 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305242 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 00:24:06.305272 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305256 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 00:24:06.305272 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305245 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 00:24:06.305519 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305208 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 00:24:06.305519 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305256 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7rnajk1nfvaac\"" Nov 01 00:24:06.305519 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305386 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j7jdw\"" Nov 01 00:24:06.305519 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.305503 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 00:24:06.323067 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.323028 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:06.355299 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.355260 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 00:24:06.355441 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.355273 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 00:24:06.355441 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.355357 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 00:24:06.356076 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.356059 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 00:24:06.357724 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.357707 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:24:06.363379 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363498 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363385 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363574 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363555 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363616 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363616 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363611 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363704 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363641 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363750 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363720 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363806 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363772 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363853 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363803 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363853 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363945 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g85t\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363945 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363906 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.363945 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.364073 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.364073 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.363998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.364165 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.364088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.364165 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.364113 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.364165 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.364157 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.364506 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.364163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.364506 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.364187 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.364506 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.364217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.365046 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.364646 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.365046 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.364740 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.367059 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.365771 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.367059 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.365796 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.367059 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.366024 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.367059 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.366616 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.367059 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.367002 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.367975 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.367743 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.368202 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.368177 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.368351 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.368332 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.368678 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.368655 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.369009 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.368988 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.369110 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.369091 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.369215 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.369192 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.369967 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.369943 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.370127 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.370108 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.370170 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.370130 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.370390 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.370371 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.425995 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.425962 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g85t\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:06.582685 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:06.582648 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:07.020689 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.020608 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:24:07.023709 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:07.023683 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186510bd_08db_41ac_9e29_4907eb505d01.slice/crio-b8a27a8342fc5c9c161f80120b904e045b4ad2f50dd27b7f0e8b05daaac643a4 WatchSource:0}: Error finding container b8a27a8342fc5c9c161f80120b904e045b4ad2f50dd27b7f0e8b05daaac643a4: Status 404 returned error can't find the container with id b8a27a8342fc5c9c161f80120b904e045b4ad2f50dd27b7f0e8b05daaac643a4 Nov 01 00:24:07.575560 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.575526 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:07.575773 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.575609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:24:07.627380 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.627346 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 00:24:07.627456 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.627405 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 00:24:07.638841 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.638806 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:07.638841 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.638830 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:24:07.676865 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.676830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:24:07.728545 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.728510 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 00:24:07.788855 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.788821 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 00:24:07.800489 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.800464 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:24:07.855867 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.855781 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rjfbh\"" Nov 01 00:24:07.860840 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.860820 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7ckpz\"" Nov 01 00:24:07.865057 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.865015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:07.870159 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.870141 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:24:07.961070 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.961035 2570 generic.go:356] "Generic (PLEG): container finished" podID="186510bd-08db-41ac-9e29-4907eb505d01" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" exitCode=0 Nov 01 00:24:07.961225 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.961084 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerDied","Data":"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4"} Nov 01 00:24:07.961225 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:07.961154 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerStarted","Data":"b8a27a8342fc5c9c161f80120b904e045b4ad2f50dd27b7f0e8b05daaac643a4"} Nov 01 00:24:08.150817 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:08.150742 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t74k4\"" Nov 01 00:24:08.159926 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:08.159882 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:24:08.239386 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:08.239358 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4r6x"] Nov 01 00:24:08.242083 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:08.242053 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f953789_9f33_475b_9687_714c6960973c.slice/crio-dbfb9dd3144f2d606524349542fe225de3f1f043bded0d4120cb3aa97b3a43e2 WatchSource:0}: Error finding container dbfb9dd3144f2d606524349542fe225de3f1f043bded0d4120cb3aa97b3a43e2: Status 404 returned error can't find the container with id dbfb9dd3144f2d606524349542fe225de3f1f043bded0d4120cb3aa97b3a43e2 Nov 01 00:24:08.255049 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:08.254994 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2mlf"] Nov 01 00:24:08.257336 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:08.257310 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c3c93f_5487_4bbe_a237_4528983d8d38.slice/crio-1326afeaf1284735b36e7a9b96f9dd23d3d297d4b062221f7c0db3fe14911f71 WatchSource:0}: Error finding container 1326afeaf1284735b36e7a9b96f9dd23d3d297d4b062221f7c0db3fe14911f71: Status 404 returned error can't find the container with id 1326afeaf1284735b36e7a9b96f9dd23d3d297d4b062221f7c0db3fe14911f71 Nov 01 00:24:08.538204 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:08.538146 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9nfbp"] Nov 01 00:24:08.540552 ip-10-0-141-93 kubenswrapper[2570]: W1101 00:24:08.540527 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf719590_ce53_4a44_af67_05c687f68258.slice/crio-3c6f951c5105104e2ce8b8006b0080433592c8d8719eb887370d78890cf0970e WatchSource:0}: Error finding container 3c6f951c5105104e2ce8b8006b0080433592c8d8719eb887370d78890cf0970e: Status 404 returned error can't find the container with id 3c6f951c5105104e2ce8b8006b0080433592c8d8719eb887370d78890cf0970e Nov 01 00:24:08.967849 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:08.967760 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2mlf" event={"ID":"15c3c93f-5487-4bbe-a237-4528983d8d38","Type":"ContainerStarted","Data":"1326afeaf1284735b36e7a9b96f9dd23d3d297d4b062221f7c0db3fe14911f71"} Nov 01 00:24:08.969088 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:08.969064 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9nfbp" event={"ID":"bf719590-ce53-4a44-af67-05c687f68258","Type":"ContainerStarted","Data":"3c6f951c5105104e2ce8b8006b0080433592c8d8719eb887370d78890cf0970e"} Nov 01 00:24:08.970415 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:08.970385 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4r6x" event={"ID":"5f953789-9f33-475b-9687-714c6960973c","Type":"ContainerStarted","Data":"dbfb9dd3144f2d606524349542fe225de3f1f043bded0d4120cb3aa97b3a43e2"} Nov 01 00:24:09.009570 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:09.009532 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:09.211855 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:09.211766 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:09.417764 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:09.417726 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:10.937896 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:10.937848 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:10.958686 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:10.958659 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:10.987403 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:10.987362 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2mlf" event={"ID":"15c3c93f-5487-4bbe-a237-4528983d8d38","Type":"ContainerStarted","Data":"0990473f863fdf13923fa46f47265adc207e741a84b578d9b1275871ef0818ae"} Nov 01 00:24:10.987582 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:10.987427 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2mlf" event={"ID":"15c3c93f-5487-4bbe-a237-4528983d8d38","Type":"ContainerStarted","Data":"35f56695ac397c83f405fc2a9a8a927a12d016fee7c4648588ee4f13598b65de"} Nov 01 00:24:11.156936 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:11.156895 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:11.302451 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:11.302384 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v2mlf" podStartSLOduration=66.666975963 podStartE2EDuration="1m8.302367197s" totalImagesPullingTime="1.635391234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:11.30118361 +0000 UTC m=+69.177794995" watchObservedRunningTime="2026-11-01 00:24:11.302367197 +0000 UTC m=+69.178978582" Nov 01 00:24:11.921426 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:11.921389 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:12.132939 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:12.132909 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:12.714714 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:12.713334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:12.998417 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:12.998382 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:13.011966 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:13.011926 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerStarted","Data":"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643"} Nov 01 00:24:13.012091 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:13.012028 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerStarted","Data":"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1"} Nov 01 00:24:13.012091 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:13.012080 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerStarted","Data":"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b"} Nov 01 00:24:13.012207 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:13.012125 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerStarted","Data":"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c"} Nov 01 00:24:13.012207 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:13.012168 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerStarted","Data":"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415"} Nov 01 00:24:13.950099 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:13.950069 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:13.950500 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:13.950480 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:13.957092 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:13.957069 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:14.019477 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.019404 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9nfbp" event={"ID":"bf719590-ce53-4a44-af67-05c687f68258","Type":"ContainerStarted","Data":"3c052c3dc0bd18dccb9356ec3658892bebb6127d5dfb87ca382f278879bd3ecb"} Nov 01 00:24:14.019643 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.019484 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:24:14.023886 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.023847 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerStarted","Data":"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065"} Nov 01 00:24:14.028565 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.028541 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:14.309552 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.309487 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9nfbp" podStartSLOduration=65.950463341 podStartE2EDuration="1m11.309465212s" totalImagesPullingTime="5.359001871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:14.190689569 +0000 UTC m=+72.067300954" watchObservedRunningTime="2026-11-01 00:24:14.309465212 +0000 UTC m=+72.186076597" Nov 01 00:24:14.339553 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.339495 2570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5665b49b4d-hvmbv"] Nov 01 00:24:14.444190 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.444130 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.886441739 podStartE2EDuration="8.444111094s" totalImagesPullingTime="4.540927109s" totalInitContainerRuntime="16.742246ms" isStatefulPod=false podCreationTimestamp="2026-11-01 00:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:14.4426189 +0000 UTC m=+72.319230286" watchObservedRunningTime="2026-11-01 00:24:14.444111094 +0000 UTC m=+72.320722483" Nov 01 00:24:14.648887 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.648797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:14.890986 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:14.890929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:15.029617 ip-10-0-141-93 kubenswrapper[2570]: I1101 00:24:15.029522 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4r6x" event={"ID":"5f953789-9f33-475b-9687-714c6960973c","Type":"ContainerStarted","Data":"5492a8d90a8844ffa2a0d169b6e44ddf58a5390706b2d1179dec144f13a1b7d1"} Nov 01 00:24:15.061128 ip-10-0-141-93 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 00:24:15.070036 ip-10-0-141-93 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 00:24:15.070212 ip-10-0-141-93 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 00:24:15.070265 ip-10-0-141-93 systemd[1]: kubelet.service: Consumed 5.088s CPU time. Nov 01 00:24:15.087368 ip-10-0-141-93 systemd[1]: Starting Kubernetes Kubelet... Nov 01 00:24:15.157806 ip-10-0-141-93 kubenswrapper[6036]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:24:15.157806 ip-10-0-141-93 kubenswrapper[6036]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 00:24:15.157806 ip-10-0-141-93 kubenswrapper[6036]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:24:15.157806 ip-10-0-141-93 kubenswrapper[6036]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:24:15.160014 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.159922 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:24:15.160014 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160014 6036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:24:15.160014 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160018 6036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160021 6036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160024 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160027 6036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160030 6036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160033 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160035 6036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160038 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160041 6036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160043 6036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160046 6036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160049 6036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160052 6036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160055 6036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160057 6036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160060 6036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160063 6036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160065 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160068 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160070 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:24:15.160112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160073 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160076 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160079 6036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160082 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160084 6036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160087 6036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160090 6036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160093 6036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160095 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160098 6036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160101 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160104 6036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160106 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160111 6036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160116 6036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160120 6036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160124 6036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160128 6036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160131 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:24:15.160639 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160134 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160137 6036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160140 6036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160143 6036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160145 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160148 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160151 6036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160153 6036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160156 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160159 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160162 6036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160164 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160167 6036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160169 6036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160173 6036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160175 6036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160179 6036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160182 6036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160184 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:24:15.161079 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160187 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160189 6036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160192 6036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160194 6036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160196 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160199 6036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160201 6036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160204 6036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160206 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160209 6036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160211 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160214 6036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160216 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160219 6036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160221 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160224 6036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160227 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160230 6036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160232 6036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160235 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:24:15.161543 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160238 6036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160241 6036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160244 6036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160247 6036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160249 6036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160251 6036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160254 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160257 6036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160259 6036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160262 6036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160264 6036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160267 6036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160269 6036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160272 6036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160274 6036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160277 6036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160295 6036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160300 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160304 6036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160308 6036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:24:15.162006 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160312 6036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160315 6036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160317 6036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160321 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160324 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.160327 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161107 6036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161113 6036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161116 6036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161121 6036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161124 6036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161126 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161129 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161132 6036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161135 6036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161137 6036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161140 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161142 6036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161145 6036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161147 6036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:24:15.162483 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161150 6036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161154 6036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161158 6036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161161 6036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161164 6036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161167 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161169 6036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161173 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161175 6036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161178 6036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161180 6036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161183 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161185 6036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161188 6036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161191 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161193 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161196 6036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161200 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161203 6036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:24:15.162954 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161207 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161209 6036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161212 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161216 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161219 6036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161221 6036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161224 6036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161227 6036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161229 6036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161232 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161234 6036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161237 6036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161239 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161242 6036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161245 6036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161248 6036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161250 6036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161253 6036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161255 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161258 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:24:15.163413 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161260 6036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161263 6036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161265 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161268 6036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161270 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161273 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161275 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161277 6036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161301 6036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161305 6036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161307 6036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161310 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161313 6036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161315 6036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161318 6036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161320 6036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161331 6036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161334 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161336 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161339 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:24:15.163890 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161341 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161344 6036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161346 6036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161349 6036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161359 6036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161362 6036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161365 6036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161368 6036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161371 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161373 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161376 6036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161378 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161381 6036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161384 6036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161386 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161388 6036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161391 6036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161393 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161396 6036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:24:15.164381 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161398 6036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161401 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161403 6036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161406 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161409 6036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161411 6036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161414 6036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161417 6036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161420 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161422 6036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161425 6036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161428 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161431 6036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161433 6036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161952 6036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161957 6036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161960 6036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161963 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161965 6036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:24:15.164828 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161968 6036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161971 6036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161974 6036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161976 6036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161979 6036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161981 6036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161983 6036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161986 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161989 6036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161991 6036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161993 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161996 6036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.161999 6036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162003 6036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162007 6036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162010 6036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162013 6036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162016 6036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162018 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:24:15.165261 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162021 6036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162023 6036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162026 6036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162028 6036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162037 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162040 6036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162042 6036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162045 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162048 6036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162050 6036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162053 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162055 6036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162058 6036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162061 6036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162064 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162066 6036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162069 6036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162072 6036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162074 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162077 6036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:24:15.165756 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162079 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162081 6036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162084 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162087 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162090 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162092 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162095 6036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162097 6036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162100 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162102 6036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162105 6036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162108 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162110 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162113 6036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162116 6036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162118 6036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162121 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162130 6036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162134 6036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162136 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:24:15.166249 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162139 6036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162141 6036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162144 6036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162146 6036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162148 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162154 6036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162156 6036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162159 6036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162162 6036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162164 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162166 6036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162170 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162173 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162176 6036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162178 6036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162181 6036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162184 6036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162186 6036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162189 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162191 6036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:24:15.166724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162193 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162196 6036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162199 6036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162201 6036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162203 6036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162206 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162209 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162211 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162213 6036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162216 6036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162224 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162227 6036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162229 6036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162232 6036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162234 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162236 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162239 6036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162243 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162245 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:24:15.167204 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162248 6036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162250 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.162255 6036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162377 6036 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162391 6036 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162397 6036 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162402 6036 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162406 6036 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162410 6036 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162428 6036 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162434 6036 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162437 6036 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162440 6036 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162444 6036 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162447 6036 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162450 6036 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162453 6036 flags.go:64] FLAG: --cgroup-root="" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162456 6036 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162459 6036 flags.go:64] FLAG: --client-ca-file="" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162462 6036 flags.go:64] FLAG: --cloud-provider="external" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162465 6036 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162473 6036 flags.go:64] FLAG: --cluster-domain="" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162476 6036 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 00:24:15.167665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162479 6036 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162488 6036 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162492 6036 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162497 6036 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162500 6036 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162503 6036 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162506 6036 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162510 6036 flags.go:64] FLAG: --contention-profiling="false" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162513 6036 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162516 6036 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162519 6036 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162522 6036 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162530 6036 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162534 6036 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162537 6036 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162540 6036 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162543 6036 flags.go:64] FLAG: --enable-server="true" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162546 6036 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162553 6036 flags.go:64] FLAG: --event-burst="100" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162556 6036 flags.go:64] FLAG: --event-qps="50" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162559 6036 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162562 6036 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162565 6036 flags.go:64] FLAG: --eviction-hard="" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162569 6036 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 00:24:15.168205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162572 6036 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162576 6036 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162579 6036 flags.go:64] FLAG: --eviction-soft="" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162582 6036 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162584 6036 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162587 6036 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162590 6036 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162593 6036 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162596 6036 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162599 6036 flags.go:64] FLAG: --feature-gates="" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162608 6036 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162612 6036 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162615 6036 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162618 6036 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162621 6036 flags.go:64] FLAG: --healthz-port="10248" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162624 6036 flags.go:64] FLAG: --help="false" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162626 6036 flags.go:64] FLAG: --hostname-override="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162630 6036 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162633 6036 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162636 6036 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162639 6036 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162644 6036 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162647 6036 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 00:24:15.168791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162650 6036 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162653 6036 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162656 6036 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162659 6036 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162662 6036 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162665 6036 flags.go:64] FLAG: --kube-reserved="" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162667 6036 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162670 6036 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162673 6036 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162676 6036 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162678 6036 flags.go:64] FLAG: --lock-file="" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162681 6036 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162684 6036 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162687 6036 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162693 6036 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162696 6036 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162699 6036 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162702 6036 flags.go:64] FLAG: --logging-format="text" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162704 6036 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162708 6036 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162722 6036 flags.go:64] FLAG: --manifest-url="" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162725 6036 flags.go:64] FLAG: --manifest-url-header="" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162730 6036 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162733 6036 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162736 6036 flags.go:64] FLAG: --max-pods="110" Nov 01 00:24:15.169342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162740 6036 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162742 6036 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162745 6036 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162748 6036 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162751 6036 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162755 6036 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162758 6036 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162765 6036 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162768 6036 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162771 6036 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162774 6036 flags.go:64] FLAG: --pod-cidr="" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162777 6036 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162780 6036 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162782 6036 flags.go:64] FLAG: --pods-per-core="0" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162785 6036 flags.go:64] FLAG: --port="10250" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162788 6036 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162791 6036 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-015c297573811e9b6" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162794 6036 flags.go:64] FLAG: --qos-reserved="" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162797 6036 flags.go:64] FLAG: --read-only-port="10255" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162800 6036 flags.go:64] FLAG: --register-node="true" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162803 6036 flags.go:64] FLAG: --register-with-taints="" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162807 6036 flags.go:64] FLAG: --registry-burst="10" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162810 6036 flags.go:64] FLAG: --registry-qps="5" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162813 6036 flags.go:64] FLAG: --reserved-cpus="" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162815 6036 flags.go:64] FLAG: --reserved-memory="" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162819 6036 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 00:24:15.169923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162822 6036 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162825 6036 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162833 6036 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162836 6036 flags.go:64] FLAG: --runonce="false" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162840 6036 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162843 6036 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162846 6036 flags.go:64] FLAG: --seccomp-default="false" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162849 6036 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162851 6036 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162854 6036 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162857 6036 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162862 6036 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162865 6036 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162868 6036 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162871 6036 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162874 6036 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162877 6036 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162880 6036 flags.go:64] FLAG: --system-cgroups="" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162883 6036 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162888 6036 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162891 6036 flags.go:64] FLAG: --tls-cert-file="" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162893 6036 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162900 6036 flags.go:64] FLAG: --tls-min-version="" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162903 6036 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162906 6036 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 00:24:15.170724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162909 6036 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162912 6036 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162916 6036 flags.go:64] FLAG: --v="2" Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162920 6036 flags.go:64] FLAG: --version="false" Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162925 6036 flags.go:64] FLAG: --vmodule="" Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162928 6036 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.162931 6036 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163098 6036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163103 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163106 6036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163109 6036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163112 6036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163114 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163117 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163120 6036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163122 6036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163125 6036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163127 6036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163131 6036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163134 6036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163136 6036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:24:15.171822 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163139 6036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163141 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163145 6036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163149 6036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163152 6036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163155 6036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163157 6036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163160 6036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163162 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163165 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163167 6036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163170 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163173 6036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163176 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163178 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163181 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163183 6036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163186 6036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163188 6036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163191 6036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:24:15.172481 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163193 6036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163196 6036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163199 6036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163201 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163204 6036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163206 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163209 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163212 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163214 6036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163218 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163220 6036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163223 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163226 6036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163228 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163231 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163233 6036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163236 6036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163238 6036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163241 6036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:24:15.172953 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163243 6036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163246 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163248 6036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163251 6036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163254 6036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163257 6036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163260 6036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163262 6036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163265 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163268 6036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163270 6036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163273 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163275 6036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163278 6036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163298 6036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163301 6036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163305 6036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163307 6036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163310 6036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:24:15.173498 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163312 6036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163315 6036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163318 6036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163321 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163324 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163327 6036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163329 6036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163332 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163334 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163337 6036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163339 6036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163342 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163344 6036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163346 6036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163349 6036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163352 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163354 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163356 6036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163360 6036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:24:15.173963 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163362 6036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163366 6036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163370 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163373 6036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163376 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163379 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163381 6036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163383 6036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163386 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163388 6036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163391 6036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163394 6036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163396 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163399 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163401 6036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:24:15.174588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.163428 6036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163473 6036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163478 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163481 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163484 6036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163487 6036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163490 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163492 6036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163496 6036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163498 6036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163501 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163504 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163507 6036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163509 6036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163512 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163514 6036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163518 6036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163521 6036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163523 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163526 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:24:15.175005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163529 6036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163531 6036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163533 6036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163536 6036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163538 6036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163541 6036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163543 6036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163546 6036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163548 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163552 6036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163557 6036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163560 6036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163562 6036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163565 6036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163569 6036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163571 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163574 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163577 6036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163579 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163582 6036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:24:15.175866 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163584 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163587 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163590 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163592 6036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163595 6036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163597 6036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163600 6036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163602 6036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163606 6036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163608 6036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163611 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163613 6036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163616 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163618 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163621 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163623 6036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163626 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163628 6036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163631 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163633 6036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:24:15.176759 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163635 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163638 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163641 6036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163643 6036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163646 6036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163648 6036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163651 6036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163653 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163655 6036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163659 6036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163662 6036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163664 6036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163666 6036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163669 6036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163671 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163674 6036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163676 6036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163678 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163681 6036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:24:15.177634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163683 6036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163688 6036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163691 6036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163694 6036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163697 6036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163699 6036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163702 6036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163704 6036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163707 6036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163709 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163712 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163714 6036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163717 6036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163719 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163722 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163724 6036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163727 6036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163730 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163732 6036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:24:15.178192 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163735 6036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163737 6036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163740 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163742 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163744 6036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163747 6036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163749 6036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163754 6036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.163757 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.167728 6036 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.167747 6036 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.167764 6036 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.167769 6036 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167833 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167838 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167842 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167845 6036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167847 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167851 6036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167854 6036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:24:15.178729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167857 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167859 6036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167862 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167865 6036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167867 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167870 6036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167873 6036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167876 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167879 6036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167882 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167885 6036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167889 6036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167893 6036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167896 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167898 6036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167901 6036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167904 6036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167906 6036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167909 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:24:15.179269 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167912 6036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167914 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167917 6036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167925 6036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167928 6036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167931 6036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167933 6036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167936 6036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167939 6036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167941 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167944 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167946 6036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167949 6036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167952 6036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167954 6036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167957 6036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167959 6036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167962 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167964 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:24:15.179781 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167967 6036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167969 6036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167972 6036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167975 6036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167978 6036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167980 6036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167984 6036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167986 6036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167989 6036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167992 6036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167994 6036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167997 6036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.167999 6036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168002 6036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168005 6036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168007 6036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168010 6036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168013 6036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168015 6036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168018 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:24:15.180231 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168020 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168023 6036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168026 6036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168028 6036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168031 6036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168033 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168036 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168038 6036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168040 6036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168043 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168045 6036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168048 6036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168050 6036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168053 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168055 6036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168058 6036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168061 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168064 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168067 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:24:15.180730 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168069 6036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168072 6036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168074 6036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168077 6036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168079 6036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168082 6036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168084 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168087 6036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168089 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168092 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168094 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168097 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168100 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168103 6036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168105 6036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168108 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168110 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168114 6036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168118 6036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:24:15.181218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168120 6036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168123 6036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168125 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.168152 6036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168336 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168342 6036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168345 6036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168348 6036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168352 6036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168355 6036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168357 6036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168360 6036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168363 6036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168365 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168368 6036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:24:15.181731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168370 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168373 6036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168375 6036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168378 6036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168380 6036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168384 6036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168386 6036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168389 6036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168392 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168394 6036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168397 6036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168400 6036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168403 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168405 6036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168408 6036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168410 6036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168412 6036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168415 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168417 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168420 6036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:24:15.182139 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168422 6036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168425 6036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168427 6036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168430 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168432 6036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168435 6036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168438 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168440 6036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168444 6036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168448 6036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168451 6036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168454 6036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168457 6036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168459 6036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168461 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168464 6036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168466 6036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168468 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168471 6036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168474 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:24:15.182636 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168476 6036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168479 6036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168482 6036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168484 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168487 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168489 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168492 6036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168494 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168497 6036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168499 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168501 6036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168504 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168507 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168509 6036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168512 6036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168514 6036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168516 6036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168519 6036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168522 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:24:15.183131 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168524 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168526 6036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168529 6036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168532 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168534 6036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168537 6036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168539 6036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168541 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168544 6036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168546 6036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168549 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168551 6036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168554 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168556 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168559 6036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168562 6036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168566 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168569 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168572 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:24:15.183641 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168575 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168578 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168581 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168583 6036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168586 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168588 6036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168591 6036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168593 6036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168596 6036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168598 6036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168601 6036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168603 6036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168606 6036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168609 6036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168611 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168613 6036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:24:15.184101 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168616 6036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.168639 6036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168671 6036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168675 6036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168677 6036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168680 6036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168683 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168685 6036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168688 6036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168690 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168693 6036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168696 6036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168700 6036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168703 6036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168706 6036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168709 6036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:24:15.184549 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168712 6036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168714 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168717 6036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168719 6036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168722 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168724 6036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168727 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168729 6036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168732 6036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168734 6036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168737 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168742 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168744 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168746 6036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168749 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168751 6036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168754 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168756 6036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168758 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168761 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:24:15.184978 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168763 6036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168766 6036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168768 6036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168770 6036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168773 6036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168775 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168778 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168780 6036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168782 6036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168785 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168788 6036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168790 6036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168793 6036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168796 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168798 6036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168803 6036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168805 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168808 6036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168810 6036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:24:15.185475 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168813 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168816 6036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168820 6036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168824 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168828 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168830 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168833 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168836 6036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168839 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168842 6036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168844 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168847 6036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168849 6036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168852 6036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168855 6036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168857 6036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168860 6036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168862 6036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168865 6036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:24:15.185936 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168867 6036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168869 6036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168872 6036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168874 6036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168877 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168879 6036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168881 6036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168884 6036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168887 6036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168891 6036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168893 6036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168896 6036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168898 6036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168901 6036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168903 6036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168906 6036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168908 6036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168911 6036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168914 6036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:24:15.186417 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168916 6036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168919 6036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168921 6036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168924 6036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168927 6036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168929 6036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168932 6036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168934 6036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168937 6036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168939 6036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168942 6036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168944 6036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168947 6036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168949 6036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:15.168952 6036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.169009 6036 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.170039 6036 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.170102 6036 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.170298 6036 server.go:1014] "Starting client certificate rotation" Nov 01 00:24:15.186882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.171431 6036 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 00:24:15.187463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.171597 6036 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 00:24:15.187463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.171660 6036 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 00:18:02 +0000 UTC" deadline="2028-07-02 23:52:38.222843245 +0000 UTC" Nov 01 00:24:15.187463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.171707 6036 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14639h28m23.051140107s" Nov 01 00:24:15.187463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.173403 6036 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 00:24:15.187463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.174623 6036 log.go:25] "Validated CRI v1 runtime API" Nov 01 00:24:15.187463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.177117 6036 log.go:25] "Validated CRI v1 image API" Nov 01 00:24:15.187463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.177900 6036 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 00:24:15.187463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.180462 6036 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7b9b65ff-87d0-4284-b668-2a6f56018008:/dev/nvme0n1p4 d5938dd8-c895-4fb8-8d88-2abf3518cf3c:/dev/nvme0n1p3] Nov 01 00:24:15.187714 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.180487 6036 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ddc0719a785852a8f3464f1052c47bf5d7fa10df9fa06605370c2d67a77c057/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ddc0719a785852a8f3464f1052c47bf5d7fa10df9fa06605370c2d67a77c057/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1234b17a183cb5348a6d0f5a0113df6fae4857630b14d99e244027db584cca67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1234b17a183cb5348a6d0f5a0113df6fae4857630b14d99e244027db584cca67/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1326afeaf1284735b36e7a9b96f9dd23d3d297d4b062221f7c0db3fe14911f71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1326afeaf1284735b36e7a9b96f9dd23d3d297d4b062221f7c0db3fe14911f71/userdata/shm major:0 minor:491 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/207043ddccbe780b1664dc5d87a9f9f9d9cd2218fd72997089fee24aecf61db7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/207043ddccbe780b1664dc5d87a9f9f9d9cd2218fd72997089fee24aecf61db7/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36c62bd8c0d4ca31b0e2e148fb96cd70e52355b583aec8610548c1d8c6a4ef94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36c62bd8c0d4ca31b0e2e148fb96cd70e52355b583aec8610548c1d8c6a4ef94/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c6f951c5105104e2ce8b8006b0080433592c8d8719eb887370d78890cf0970e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c6f951c5105104e2ce8b8006b0080433592c8d8719eb887370d78890cf0970e/userdata/shm major:0 minor:493 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/401f1de8b558af3b8ac80748f24fc3ea813e9e99a1fbf78be70e6a08dac1624f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/401f1de8b558af3b8ac80748f24fc3ea813e9e99a1fbf78be70e6a08dac1624f/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/476c72a3effe87e598627cb59ab7085f34ffd9d23ae23bd3c0e34c6b94cd451b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/476c72a3effe87e598627cb59ab7085f34ffd9d23ae23bd3c0e34c6b94cd451b/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48eadde52d4b4d9834ec26c6765bab3ac53bad6d8ebee4aeb991c8ec2a88fd0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48eadde52d4b4d9834ec26c6765bab3ac53bad6d8ebee4aeb991c8ec2a88fd0a/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c45e0d057b00b805e16c4cb0878a08fe02ccdc83b6f33e2dda0d79876a404ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c45e0d057b00b805e16c4cb0878a08fe02ccdc83b6f33e2dda0d79876a404ea/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dce003717d516c671a1c0a1099a02b3d5547c8d88c03545c168e3156a658abe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dce003717d516c671a1c0a1099a02b3d5547c8d88c03545c168e3156a658abe/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e63241786a261f1ca96741f564820e6fa79be6772a9bded988e9610598ea232/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e63241786a261f1ca96741f564820e6fa79be6772a9bded988e9610598ea232/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/611b04993ccdb245bd8758384f55c870ccaaf227d5515750d08a34238b55db7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/611b04993ccdb245bd8758384f55c870ccaaf227d5515750d08a34238b55db7c/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/670484c9353bc5993f26641d6532bc27a214a32ef08aaef251fbda58cfc2bc80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/670484c9353bc5993f26641d6532bc27a214a32ef08aaef251fbda58cfc2bc80/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67d02bfe87504573f9473b10cb83becef096897d9004e7553b8145d0934b5bdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67d02bfe87504573f9473b10cb83becef096897d9004e7553b8145d0934b5bdb/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/686938818e898cee1468a8034c45c472a9829dce7b8043b88413fb089af4cd03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/686938818e898cee1468a8034c45c472a9829dce7b8043b88413fb089af4cd03/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68791c348fd78f0f47eea65b047ee982cc236034921c439185b168cbbc024f33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68791c348fd78f0f47eea65b047ee982cc236034921c439185b168cbbc024f33/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74298718edc986141ebc52073eb6b38b8b6600d8339dcb581f2fa20f2640a91a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74298718edc986141ebc52073eb6b38b8b6600d8339dcb581f2fa20f2640a91a/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80471ee0f664100ace069d095e8fe35c6294918d1ed498f111a85d172307fe30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80471ee0f664100ace069d095e8fe35c6294918d1ed498f111a85d172307fe30/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a277d0a81f7bfedcc7b14be37afbb5d413453332fba41bc021b3415c3cf6ac3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a277d0a81f7bfedcc7b14be37afbb5d413453332fba41bc021b3415c3cf6ac3/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af959519697a9d5a96ba08701cf361c58180b14c70b614c45c12cf7b81f938d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af959519697a9d5a96ba08701cf361c58180b14c70b614c45c12cf7b81f938d6/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8a27a8342fc5c9c161f80120b904e045b4ad2f50dd27b7f0e8b05daaac643a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8a27a8342fc5c9c161f80120b904e045b4ad2f50dd27b7f0e8b05daaac643a4/userdata/shm major:0 minor:482 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba4f0543a9d2da11d38ed09e342bf56da79f0291f08c95373d2362cb63c40865/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba4f0543a9d2da11d38ed09e342bf56da79f0291f08c95373d2362cb63c40865/userdata/shm major:0 minor:402 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce2a983f542730942e817f2aab8cc51ef2c45b446a1dbd19ab7a3a7ab511c7c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce2a983f542730942e817f2aab8cc51ef2c45b446a1dbd19ab7a3a7ab511c7c8/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d568f7d494cf16f683795b12308fa36422f85d82cd5de081c8ae4cdfa2139186/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d568f7d494cf16f683795b12308fa36422f85d82cd5de081c8ae4cdfa2139186/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbfb9dd3144f2d606524349542fe225de3f1f043bded0d4120cb3aa97b3a43e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbfb9dd3144f2d606524349542fe225de3f1f043bded0d4120cb3aa97b3a43e2/userdata/shm major:0 minor:489 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e52c8c3ee9469ec090cc1711c535529904b8edf292ca17580df2f784d2ddcfa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e52c8c3ee9469ec090cc1711c535529904b8edf292ca17580df2f784d2ddcfa4/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/alertmanager/9 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/config-reloader/6 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~projected/kube-api-access-r2q7j:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~projected/kube-api-access-r2q7j major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~projected/tls-assets major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/config-volume major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/web-config major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15c3c93f-5487-4bbe-a237-4528983d8d38/volumes/kubernetes.io~projected/kube-api-access-jkcbn:{mountpoint:/var/lib/kubelet/pods/15c3c93f-5487-4bbe-a237-4528983d8d38/volumes/kubernetes.io~projected/kube-api-access-jkcbn major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15c3c93f-5487-4bbe-a237-4528983d8d38/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/15c3c93f-5487-4bbe-a237-4528983d8d38/volumes/kubernetes.io~secret/metrics-certs major:0 minor:487 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/config-reloader/2 major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/prometheus/15 major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~empty-dir/config-out major:0 minor:480 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~projected/kube-api-access-5g85t:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~projected/kube-api-access-5g85t major:0 minor:481 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~projected/tls-assets major:0 minor:472 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/config major:0 minor:476 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:478 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:477 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:479 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:473 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:475 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/web-config major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~projected/kube-api-access-8x5g6:{mountpoint:/var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~projected/kube-api-access-8x5g6 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~projected/kube-api-access-cnxjh:{mountpoint:/var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~projected/kube-api-access-cnxjh major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34995d02-7971-4439-9d1c-c1083cc72be9/volumes/kubernetes.io~projected/kube-api-access-crwwm:{mountpoint:/var/lib/kubelet/pods/34995d02-7971-4439-9d1c-c1083cc72be9/volumes/kubernetes.io~projected/kube-api-access-crwwm major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34995d02-7971-4439-9d1c-c1083cc72be9/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/34995d02-7971-4439-9d1c-c1083cc72be9/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e567114-6e47-4337-9776-1ec7f47be0b8/volumes/kubernetes.io~projected/kube-api-access-29zvl:{mountpoint:/var/lib/kubelet/pods/3e567114-6e47-4337-9776-1ec7f47be0b8/volumes/kubernetes.io~projected/kube-api-access-29zvl major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f/volumes/kubernetes.io~projected/kube-api-access-xf7x9:{mountpoint:/var/lib/kubelet/pods/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f/volumes/kubernetes.io~projected/kube-api-access-xf7x9 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f953789-9f33-475b-9687-714c6960973c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5f953789-9f33-475b-9687-714c6960973c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:486 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~projected/kube-api-access-hf6qp:{mountpoint:/var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~projected/kube-api-access-hf6qp major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef87adc-7459-49ed-9810-4482be252165/volumes/kubernetes.io~projected/kube-api-access-cnxlh:{mountpoint:/var/lib/kubelet/pods/6ef87adc-7459-49ed-9810-4482be252165/volumes/kubernetes.io~projected/kube-api-access-cnxlh major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~projected/kube-api-access-bq9g6:{mountpoint:/var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~projected/kube-api-access-bq9g6 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ad3371f-f985-4f80-b26f-d69319f24cac/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7ad3371f-f985-4f80-b26f-d69319f24cac/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80ea826e-c81c-4429-9a5d-131a2113fcb3/volumes/kubernetes.io~projected/kube-api-access-pkxw4:{mountpoint:/var/lib/kubelet/pods/80ea826e-c81c-4429-9a5d-131a2113fcb3/volumes/kubernetes.io~projected/kube-api-access-pkxw4 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~projected/kube-api-access-pgkh5:{mountpoint:/var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~projected/kube-api-access-pgkh5 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~projected/kube-api-access-9wjmz:{mountpoint:/var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~projected/kube-api-access-9wjmz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf719590-ce53-4a44-af67-05c687f68258/volumes/kubernetes.io~projected/kube-api-access-495mv:{mountpoint:/var/lib/kubelet/pods/bf719590-ce53-4a44-af67-05c687f68258/volumes/kubernetes.io~projected/kube-api-access-495mv major:0 minor:488 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c154cf87-34c2-411a-bf51-a67f2cd9e1d2/volumes/kubernetes.io~projected/kube-api-access-ljc9d:{mountpoint:/var/lib/kubelet/pods/c154cf87-34c2-411a-bf51-a67f2cd9e1d2/volumes/kubernetes.io~projected/kube-api-access-ljc9d major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c154cf87-34c2-411a-bf51-a67f2cd9e1d2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c154cf87-34c2-411a-bf51-a67f2cd9e1d2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d19fab29-f58d-4668-b635-596a496154ed/volumes/kubernetes.io~projected/kube-api-access-tsz6b:{mountpoint:/var/lib/kubelet/pods/d19fab29-f58d-4668-b635-596a496154ed/volumes/kubernetes.io~projected/kube-api-access-tsz6b major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~projected/kube-api-access-5pjr4:{mountpoint:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~projected/kube-api-access-5pjr4 major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~projected/kube-api-access-stflp:{mountpoint:/var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~projected/kube-api-access-stflp major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6f778b-df02-4af4-9e02-0d7d40321a52/volumes/kubernetes.io~projected/kube-api-access-hvkgw:{mountpoint:/var/lib/kubelet/pods/de6f778b-df02-4af4-9e02-0d7d40321a52/volumes/kubernetes.io~projected/kube-api-access-hvkgw major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6f778b-df02-4af4-9e02-0d7d40321a52/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/de6f778b-df02-4af4-9e02-0d7d40321a52/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volumes/kubernetes.io~projected/kube-api-access-vqkjh:{mountpoint:/var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volumes/kubernetes.io~projected/kube-api-access-vqkjh major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f120a481-1b91-4db7-92c7-eebb3d7e7192/volumes/kubernetes.io~projected/kube-api-access-l82rg:{mountpoint:/var/lib/kubelet/pods/f120a481-1b91-4db7-92c7-eebb3d7e7192/volumes/kubernetes.io~projected/kube-api-access-l82rg major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f120a481-1b91-4db7-92c7-eebb3d7e7192/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f120a481-1b91-4db7-92c7-eebb3d7e7192/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa5c470e-0673-4c56-a4e8-feb9fa32b67e/volumes/kubernetes.io~projected/kube-api-access-q9wd9:{mountpoint:/var/lib/kubelet/pods/fa5c470e-0673-4c56-a4e8-feb9fa32b67e/volumes/kubernetes.io~projected/kube-api-access-q9wd9 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~projected/kube-api-access-t8ktk:{mountpoint:/var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~projected/kube-api-access-t8ktk major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:374 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/46e7c1f27afdf90f86fe7566d64d2b905b1df808e7403f6bf4ab6fb8a5da08f7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/297488811bdc7eb24c09eca70f3b559c4b982f16b311058bec1a3b7c26633152/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0fa8263c87bd3de12b5f88e23be75ddc5efbb5a47b801f69734bd21b529ff3b2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9e1fa39011fd709e834efcb720ad8dd2e7319032cc0c12ee354a96c2969c0c4b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d0856ab7f3f98fc0b848e8c77adc21b3cadd361aed181ec6a868233976c29482/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fd9fac971a83bf7588db775cf6740902765dd7c671ab291b30019b9f3e9c4df5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/a8fe45c8fe0b6cf5e62d3ce4280e71f56ce593a7745ef882043b639e9cca97ad/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/eb5239de89365281b69d281c4c5d7445dfa3273807440b73f65ace3735cf58b6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/74f153b314130981ba3aa6d798f456b85692156f6a5397a330a79f8d2f3d7c27/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/1cd9e43293a0752506f8fcb110a6e8a43ebb937615a10a3a3aa852b8d42b4769/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/b2aca813725f00168a1665ba523404aebeb815ac467b91bd9793f987bcc8554e/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/492b4ee7c85febc938cc86a46bb23c54b914d6cbdcc48895aa6f0b7147ef6772/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/71b5ed431c376baf70a32bbbfff4979b124ee478030fec5547895ddf4ebc52c0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a04ceb9aa3f44205c7791b330ab2c32ff13e021fda885a8f172e5cd8baf1d05b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e183412b14f377044aca05459c5a3964b72e65042664eb9c33d51167501e1cc0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5c2adcc7547d866a902a431b3a6ba9793a5f1c11cd78299709eb5fc83a0ece4f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/00f489a75b476c906019a33be977e3cb76cf626e78df377862f76fd7abe04b1d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8ab8380fd7cf12af1a6ede8420e58ef3cb061670fd1b183b600a232fd0b5880d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e130219103fee826f824c988c7a6e459ab01ddbd4d6eb73016215b1ab96cdc4f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b60de18ec44f1a9b75de76f6b8b2f20b0a612b7054e586a7629dc00012f4f3ac/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e0540d7ce1ab0c9bb3b3044e06f3073b4ae3a472f3a944b8abe13e6743820932/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/972544051dbcb0222e91c50fe72a886446d81d95350bce7ed5bc1f5089124fac/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/1bb2937950cf073f62478c368f90d5634f99f17537df889009433ceae115ea8d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/da9304640370ddc78b56401b7e476734cca8b938284f506e6a5bb4062ef52502/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/306bbb42c40eb9efb334efc2d64c4772983eca5a31f66aa2a76f408bcb7b36b9/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/deb31e5f7eab15bace18b72e05f05a43bd80373d78c6bfc4e3d4a7da77ffa370/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/451df5420abdb7e0647ed7c3cb40f91bc11f8c2644162d18c4ce24d232566b5e/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/0ffb0c60acc24093eff165e4c9c5acd91fcde15479bc255295f6d96d64f51ac7/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/08bfdf3e26800304b50a8b652694ea5201ae4d89c20b9fe67c0ed60f906f87e8/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/68bbf28fd036e40b6597905d7156393c7420f6724d519bc0e01428c9e894ef1b/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5efdc311b8b1ba182a4557134e5443febb63cec0304b9b1d9fd4ca2ed41e42b4/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/c28b8ba7a15ca36c882ef377e15d97d0af729faa65804fc417a87b5c994079eb/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/73862552eae981cd45d00bd168d3bae46b6e440f4b9d3c0e6b380409e323672d/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/e04c2d50db5aa2de26f0538ccddf11c951f2fca9790951c808ae8c6ab1c25032/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/2384952fccdf616276ad19459e3710ca6178cb974812bf3e1fbdcd3c73e183e3/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/527380a2fdb7c05d140a694de982c1d970cceb118c70614bb35411e878ce3f05/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/4198298bed8f5ab53487656b25dee960e076880c172b75764657985e6e4abe7b/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/90813e464966560873e4b154584b45f5ec0bee8b3fc496a55cc3b367bae3631a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/45498d72340458a989f0ab8c00708249bbbd82f5b63305560d76ea70e68441f0/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/d94bc23074251f79b0c154863c3f0014b55680411f429832971c4ffbfa08f290/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/cf20de334dd56527763724bfa365c14079ca018025347a1ee36ae44427d25218/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/419ffa99aba943e1e7bf1a1785e35a69dd4a4b1e37d1aa04b8e1184962e9df9c/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/67c50e8f9e02ac53f385f7809538fab8aa66e5613fdb423311e622151ebdb61a/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/a7e1520e226725a77a91fe8147ad3bc3e815be54d3707676ca30b2617821e05a/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/7dc0baa469d990bff466e6454e515ffe90369edddf59ac5db7bbe51e7e40ad1b/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/e22a3a472fd50d651223e531fe52bc8ebee55a1a4de4bd8656cdb6484003a238/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/ee9f6b83494498ee1ccf3c99fa8b5036d058bffe52cb93c4557568f8ad68f21e/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/c30d6344865d6eee8518a8806755d05799c5beb0f6dc34ff671177d9af14ec08/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/5e245aee9ddd631fc97a9a4424ca547fa26f42499ee2038e0d66d226241c6b5f/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/008aa1b3571942822a1c6cd92e3c96d28687b0a34b1cd046e0546f8a98f619ff/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/446bcf281b0093b37076a5c2dc4ebda62c6012f80b05674a98bbb4654603c646/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/f56afe7224e4ed0e8ab2b6fcafad84a1d53371b23c1772391479c708c33e5af1/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/a3dc2be9f8f43334ba802070fef9516740a90a2e763f151601a6260a30c088c6/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/8c7f18c295596524816e9c8d9d040468d4599d2c40a69f7a6feef74f50a2baae/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/1a220624c07b79665bf996389b4078362582aba08bbd3bee9b1f28f639910ec8/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/fa8e50fbb58dd90e1d9c06fd31c8ba4c176f84c729611faeb9f2531253be32da/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/711a2ca73fcb6f6580a681c8df146912e1d95cd460d046cc3a9c909f8a1eb710/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/f609f998438f7381c9f7edfea1067389e0ba4ea10d024be68d46e422b1880cb3/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/46dab1b37880ca05443b2691473328f357558c36060a1723ee36f8e38faf924f/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/f52fcc06d3bc1d5a9362d7647629686e831abcecf025d5e4e22233929a12b1b7/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/9f1e27f5a8b4f38444676646546b3bb6d2b1d31ce2d2f1eddf964f27631065c6/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/82e1ea819f0e44ea89593fd6aa2bb127916b786468c39ef38fa4c010fd197b16/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/9420e3e71a3bd11eaf06e39794d546ddc34cdf583cdd1316e00be73cb992c7be/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/15bd5afdb3fd92e8238363441f57b56cdf038545c7495a985a84cc27bf74537e/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/63eec3ae976d69d17ae03fd707f6a8f665a0d2cc0d0026ed211308634493a199/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/11728e76ff3a3d67c493900bf0ad5f08d711b08fd350377f3078da5335d86358/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/ec4d41d8c61ff1f18f031d948be5c3444d36579e67a22f471e3d367a6c1287c3/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/0e150d1794af13a591901925b2f985e5b0d57bdf17f5d91987fafa91b601ccba/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/2d1d50e8a22afbc498dff2094a9f979604c0cf6d085becb0b51508952b5a2c46/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/292dcb814103053cfd461c68700090183d67beae0d512cf3f1b21209c7d95575/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/659a0398d13afc00ce155dc02413eec566e045b7ce55fa4c32fb4f707e29ad49/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/8180a8febf95c485fba60f367f18c8f6dfab9a71816cf884e5e4b202ed5585be/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/039ee766ce0aa873bc67172a3af270610c6c4f0f1949669ddbf779fb02431eb9/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/83b294a61779a9b869300280b35235869173412c55be9acaad774c8c92d9db93/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/4abfae9d86882ea5f887d43ccc68eef0a6a8f41b031f9bf55ce02035687b0304/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-525:{mountpoint:/var/lib/containers/storage/overlay/14819b93cf61509fb5b179c01d98d2849630685383628c9efad4192e919c1e3c/merged major:0 minor:525 fsType:overlay blockSize:0} overlay_0-530:{mountpoint:/var/lib/containers/storage/overlay/dbb90f853a6757cea01d2b39d6aa111b07942b3f3d29f76f204a9b824afd05fb/merged major:0 minor:530 fsType:overlay blockSize:0} overlay_0-535:{mountpoint:/var/lib/containers/storage/overlay/50ed0764c77996b572b08ee0eed9d7015d40ed7b083049406e5a574dcdbbceda/merged major:0 minor:535 fsType:overlay blockSize:0} overlay_0-540:{mountpoint:/var/lib/containers/storage/overlay/4c83ce3556f2232dbea29a5229dc98dde9161c9cf05656b99b8b46ba12def678/merged major:0 minor:540 fsType:overlay blockSize:0} overlay_0-545:{mountpoint:/var/lib/containers/storage/overlay/e60ce52700e9cadfd17fc2a82651e50396af487534a8def4724491d7478c3585/merged major:0 minor:545 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/5ba9f51238baef365629ba5e0c0e132a27339251aee9e41e6e62d80e25192ebe/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/1c216c9fc68a8461deb212783042138e908e4539163c93de7c97af9807b8bc17/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/06d09d6077641193268d1940f8a60f23fa756d0e806c9c2e69474979e13d0546/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/afc155b20986fe78fa2462ed2d16c3f74594adfc207b76376790ab293ed1a057/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c6c67bd5d1a1b286eb9240221111a43b9f30adca33573a80dcdb93a63e1d32fd/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4bfef7365834b6fc2f547a0cd542869e88b2649fe4aae5d0739c10ccf8c05864/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ede549ac4a8a5631f9415bdf6b875730cf646694d84bb3fad117a5d45d08c85a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/923f6209f346baf0466f51dd70e23a64f41f8ed7260618bbd252d64bdf61040a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/47039080895a313ea7592703fa65c771024cd6e386f4500af0c3a0b57b0ee243/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7fea33eda71a894b2a88c7d62a7a4fbfd89e143b229c5092156a6ac6478a1cdc/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 00:24:15.189574 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.189041 6036 manager.go:217] Machine: {Timestamp:2026-11-01 00:24:15.188548949 +0000 UTC m=+0.063769954 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103994 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca34a5227a87bc904da69085e59fa SystemUUID:ec2ca34a-5227-a87b-c904-da69085e59fa BootID:642623d8-4185-403a-82e3-ec92f3e40cc5 Filesystems:[{Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de6f778b-df02-4af4-9e02-0d7d40321a52/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c154cf87-34c2-411a-bf51-a67f2cd9e1d2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:319 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36c62bd8c0d4ca31b0e2e148fb96cd70e52355b583aec8610548c1d8c6a4ef94/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:356 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:467 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/686938818e898cee1468a8034c45c472a9829dce7b8043b88413fb089af4cd03/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf719590-ce53-4a44-af67-05c687f68258/volumes/kubernetes.io~projected/kube-api-access-495mv DeviceMajor:0 DeviceMinor:488 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:357 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:480 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~projected/kube-api-access-5g85t DeviceMajor:0 DeviceMinor:481 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1326afeaf1284735b36e7a9b96f9dd23d3d297d4b062221f7c0db3fe14911f71/userdata/shm DeviceMajor:0 DeviceMinor:491 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80471ee0f664100ace069d095e8fe35c6294918d1ed498f111a85d172307fe30/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:467 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:374 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:475 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-525 DeviceMajor:0 DeviceMinor:525 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ddc0719a785852a8f3464f1052c47bf5d7fa10df9fa06605370c2d67a77c057/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:308 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce2a983f542730942e817f2aab8cc51ef2c45b446a1dbd19ab7a3a7ab511c7c8/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34995d02-7971-4439-9d1c-c1083cc72be9/volumes/kubernetes.io~projected/kube-api-access-crwwm DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:478 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c6f951c5105104e2ce8b8006b0080433592c8d8719eb887370d78890cf0970e/userdata/shm DeviceMajor:0 DeviceMinor:493 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de6f778b-df02-4af4-9e02-0d7d40321a52/volumes/kubernetes.io~projected/kube-api-access-hvkgw DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:370 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:471 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f120a481-1b91-4db7-92c7-eebb3d7e7192/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-535 DeviceMajor:0 DeviceMinor:535 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c154cf87-34c2-411a-bf51-a67f2cd9e1d2/volumes/kubernetes.io~projected/kube-api-access-ljc9d DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~projected/kube-api-access-pgkh5 DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-530 DeviceMajor:0 DeviceMinor:530 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~projected/kube-api-access-9wjmz DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/670484c9353bc5993f26641d6532bc27a214a32ef08aaef251fbda58cfc2bc80/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/476c72a3effe87e598627cb59ab7085f34ffd9d23ae23bd3c0e34c6b94cd451b/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba4f0543a9d2da11d38ed09e342bf56da79f0291f08c95373d2362cb63c40865/userdata/shm DeviceMajor:0 DeviceMinor:402 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f953789-9f33-475b-9687-714c6960973c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:486 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/3e567114-6e47-4337-9776-1ec7f47be0b8/volumes/kubernetes.io~projected/kube-api-access-29zvl DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74298718edc986141ebc52073eb6b38b8b6600d8339dcb581f2fa20f2640a91a/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a277d0a81f7bfedcc7b14be37afbb5d413453332fba41bc021b3415c3cf6ac3/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:400 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbfb9dd3144f2d606524349542fe225de3f1f043bded0d4120cb3aa97b3a43e2/userdata/shm DeviceMajor:0 DeviceMinor:489 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fa5c470e-0673-4c56-a4e8-feb9fa32b67e/volumes/kubernetes.io~projected/kube-api-access-q9wd9 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:320 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:476 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af959519697a9d5a96ba08701cf361c58180b14c70b614c45c12cf7b81f938d6/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8a27a8342fc5c9c161f80120b904e045b4ad2f50dd27b7f0e8b05daaac643a4/userdata/shm DeviceMajor:0 DeviceMinor:482 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dce003717d516c671a1c0a1099a02b3d5547c8d88c03545c168e3156a658abe/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d19fab29-f58d-4668-b635-596a496154ed/volumes/kubernetes.io~projected/kube-api-access-tsz6b DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c45e0d057b00b805e16c4cb0878a08fe02ccdc83b6f33e2dda0d79876a404ea/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~projected/kube-api-access-stflp DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e63241786a261f1ca96741f564820e6fa79be6772a9bded988e9610598ea232/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:372 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:477 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef87adc-7459-49ed-9810-4482be252165/volumes/kubernetes.io~projected/kube-api-access-cnxlh DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1234b17a183cb5348a6d0f5a0113df6fae4857630b14d99e244027db584cca67/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/volumes/kubernetes.io~projected/kube-api-access-t8ktk DeviceMajor:0 DeviceMinor:376 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:467 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:317 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/34995d02-7971-4439-9d1c-c1083cc72be9/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:371 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:474 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afca2ec8-21f4-4bb6-bc31-009191b47dca/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-545 DeviceMajor:0 DeviceMinor:545 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/volumes/kubernetes.io~projected/kube-api-access-8x5g6 DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:467 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:473 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7ad3371f-f985-4f80-b26f-d69319f24cac/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/15c3c93f-5487-4bbe-a237-4528983d8d38/volumes/kubernetes.io~projected/kube-api-access-jkcbn DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f/volumes/kubernetes.io~projected/kube-api-access-xf7x9 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/611b04993ccdb245bd8758384f55c870ccaaf227d5515750d08a34238b55db7c/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48eadde52d4b4d9834ec26c6765bab3ac53bad6d8ebee4aeb991c8ec2a88fd0a/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd5dd766-39a4-45c1-84e8-fa86b275f100/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80ea826e-c81c-4429-9a5d-131a2113fcb3/volumes/kubernetes.io~projected/kube-api-access-pkxw4 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/207043ddccbe780b1664dc5d87a9f9f9d9cd2218fd72997089fee24aecf61db7/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/401f1de8b558af3b8ac80748f24fc3ea813e9e99a1fbf78be70e6a08dac1624f/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67d02bfe87504573f9473b10cb83becef096897d9004e7553b8145d0934b5bdb/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f120a481-1b91-4db7-92c7-eebb3d7e7192/volumes/kubernetes.io~projected/kube-api-access-l82rg DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:350 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:319 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:351 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:353 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~projected/kube-api-access-5pjr4 DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~projected/kube-api-access-cnxjh DeviceMajor:0 DeviceMinor:401 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d568f7d494cf16f683795b12308fa36422f85d82cd5de081c8ae4cdfa2139186/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes/kubernetes.io~projected/kube-api-access-bq9g6 DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:396 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:472 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:321 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9deeffc-4dcb-489c-bbbd-f472272d3c13/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:354 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:479 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68791c348fd78f0f47eea65b047ee982cc236034921c439185b168cbbc024f33/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes/kubernetes.io~projected/kube-api-access-r2q7j DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15c3c93f-5487-4bbe-a237-4528983d8d38/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:487 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-540 DeviceMajor:0 DeviceMinor:540 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e55540c7-35bd-476f-923d-d22e528e3545/volumes/kubernetes.io~projected/kube-api-access-vqkjh DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e52c8c3ee9469ec090cc1711c535529904b8edf292ca17580df2f784d2ddcfa4/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6df8f53e-1025-4a88-8945-4055732800c6/volumes/kubernetes.io~projected/kube-api-access-hf6qp DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ddc0719a785852 MacAddress:7e:f2:ae:45:e9:88 Speed:10000 Mtu:8901} {Name:1326afeaf128473 MacAddress:ca:ed:5f:d1:b9:d8 Speed:10000 Mtu:8901} {Name:207043ddccbe780 MacAddress:5a:66:a6:0a:b2:74 Speed:10000 Mtu:8901} {Name:36c62bd8c0d4ca3 MacAddress:66:6e:87:df:1c:01 Speed:10000 Mtu:8901} {Name:3c6f951c5105104 MacAddress:42:ef:44:c9:3f:5e Speed:10000 Mtu:8901} {Name:401f1de8b558af3 MacAddress:6e:95:52:5c:97:b9 Speed:10000 Mtu:8901} {Name:476c72a3effe87e MacAddress:72:80:3a:78:8d:63 Speed:10000 Mtu:8901} {Name:48eadde52d4b4d9 MacAddress:8e:28:ae:02:c8:78 Speed:10000 Mtu:8901} {Name:686938818e898ce MacAddress:9a:fb:64:25:6a:f0 Speed:10000 Mtu:8901} {Name:80471ee0f664100 MacAddress:aa:c5:65:45:e4:04 Speed:10000 Mtu:8901} {Name:8a277d0a81f7bfe MacAddress:06:9c:af:ca:8d:de Speed:10000 Mtu:8901} {Name:af959519697a9d5 MacAddress:3e:c1:90:9b:c2:2a Speed:10000 Mtu:8901} {Name:b8a27a8342fc5c9 MacAddress:62:59:3a:72:63:e4 Speed:10000 Mtu:8901} {Name:ba4f0543a9d2da1 MacAddress:ee:17:00:9e:37:03 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:55:ce:f7:d0:69 Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:f7:65:f3:40:b9 Speed:0 Mtu:8901} {Name:ce2a983f5427309 MacAddress:7a:52:d3:9d:8c:4f Speed:10000 Mtu:8901} {Name:dbfb9dd3144f2d6 MacAddress:22:27:98:a3:80:fe Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:55:ce:f7:d0:69 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ba:82:d2:67:08:db Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:ae:eb:73:cc:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 00:24:15.189835 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.189579 6036 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 00:24:15.189835 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.189631 6036 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 00:24:15.189997 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.189968 6036 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 00:24:15.190141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190000 6036 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-93.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 00:24:15.190182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190143 6036 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 00:24:15.190182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190151 6036 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 00:24:15.190182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190166 6036 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 00:24:15.190182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190179 6036 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 00:24:15.190312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190303 6036 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 00:24:15.190499 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190489 6036 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 00:24:15.190617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190608 6036 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 00:24:15.190652 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190648 6036 kubelet.go:493] "Attempting to sync node with API server" Nov 01 00:24:15.190681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190662 6036 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 00:24:15.190681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190672 6036 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 00:24:15.190733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190682 6036 kubelet.go:405] "Adding apiserver pod source" Nov 01 00:24:15.190733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.190691 6036 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 00:24:15.192425 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.192069 6036 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 00:24:15.192425 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.192097 6036 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 00:24:15.192823 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.192802 6036 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 00:24:15.195958 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.195935 6036 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 00:24:15.195958 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.195954 6036 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 00:24:15.196128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196077 6036 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 00:24:15.196349 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196333 6036 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 00:24:15.196424 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196374 6036 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 00:24:15.196510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196499 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 00:24:15.196564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196520 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 00:24:15.196564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196531 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 00:24:15.196564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196542 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 00:24:15.196564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196552 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 00:24:15.196564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196561 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 00:24:15.196880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196571 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 00:24:15.196880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196579 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 00:24:15.196880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196590 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 00:24:15.196880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196599 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 00:24:15.196880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196613 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 00:24:15.196880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196628 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 00:24:15.196880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196665 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 00:24:15.196880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.196675 6036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 00:24:15.200377 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.200359 6036 server.go:1263] "Started kubelet" Nov 01 00:24:15.200495 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.200441 6036 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 00:24:15.200558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.200469 6036 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 00:24:15.200558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.200549 6036 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 00:24:15.201051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.201032 6036 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 00:24:15.201066 ip-10-0-141-93 systemd[1]: Started Kubernetes Kubelet. Nov 01 00:24:15.202216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.202196 6036 server.go:317] "Adding debug handlers to kubelet server" Nov 01 00:24:15.205963 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.205697 6036 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 00:24:15.206068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.206006 6036 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 00:24:15.206068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.206022 6036 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 00:24:15.206301 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.206067 6036 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 00:24:15.206739 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.206718 6036 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 00:24:15.207072 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.207028 6036 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:18:03 +0000 UTC" deadline="2028-04-28 12:15:25.418716048 +0000 UTC" Nov 01 00:24:15.207138 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.207073 6036 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13067h51m10.211645905s" Nov 01 00:24:15.208059 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.208034 6036 factory.go:55] Registering systemd factory Nov 01 00:24:15.208059 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.208057 6036 factory.go:223] Registration of the systemd container factory successfully Nov 01 00:24:15.208483 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.208462 6036 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-93.us-west-1.compute.internal\" not found" Nov 01 00:24:15.208614 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.208598 6036 factory.go:153] Registering CRI-O factory Nov 01 00:24:15.208678 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.208618 6036 factory.go:223] Registration of the crio container factory successfully Nov 01 00:24:15.208678 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.208667 6036 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 00:24:15.208784 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.208695 6036 factory.go:103] Registering Raw factory Nov 01 00:24:15.208784 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.208721 6036 manager.go:1199] Started watching for new ooms in manager Nov 01 00:24:15.209723 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.209701 6036 manager.go:319] Starting recovery of all containers Nov 01 00:24:15.220454 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220407 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55540c7-35bd-476f-923d-d22e528e3545" volumeName="kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-env-overrides" seLinuxMountContext="" Nov 01 00:24:15.220454 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220453 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbf066bd-0ff4-471d-ba92-d8cd3d6820fe" volumeName="kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-tls" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220466 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b" volumeName="kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-tls" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220481 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df8f53e-1025-4a88-8945-4055732800c6" volumeName="kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-tls" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220495 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef87adc-7459-49ed-9810-4482be252165" volumeName="kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-binary-copy" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220507 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220521 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220533 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e31e6c-2296-4e75-9eed-be8579a717b9" volumeName="kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220547 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afca2ec8-21f4-4bb6-bc31-009191b47dca" volumeName="kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220558 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5dd766-39a4-45c1-84e8-fa86b275f100" volumeName="kubernetes.io/projected/bd5dd766-39a4-45c1-84e8-fa86b275f100-kube-api-access-9wjmz" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220569 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220581 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afca2ec8-21f4-4bb6-bc31-009191b47dca" volumeName="kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-metrics-client-ca" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220592 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d19fab29-f58d-4668-b635-596a496154ed" volumeName="kubernetes.io/projected/d19fab29-f58d-4668-b635-596a496154ed-kube-api-access-tsz6b" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220603 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b" volumeName="kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220618 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220629 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e31e6c-2296-4e75-9eed-be8579a717b9" volumeName="kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220643 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f953789-9f33-475b-9687-714c6960973c" volumeName="kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret" seLinuxMountContext="" Nov 01 00:24:15.220673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220656 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-grpc-tls" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220668 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220695 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e567114-6e47-4337-9776-1ec7f47be0b8" volumeName="kubernetes.io/projected/3e567114-6e47-4337-9776-1ec7f47be0b8-kube-api-access-29zvl" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220709 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220721 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220735 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbf066bd-0ff4-471d-ba92-d8cd3d6820fe" volumeName="kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-client-ca-bundle" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220746 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef87adc-7459-49ed-9810-4482be252165" volumeName="kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220759 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbf066bd-0ff4-471d-ba92-d8cd3d6820fe" volumeName="kubernetes.io/projected/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-kube-api-access-t8ktk" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220771 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220783 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15c3c93f-5487-4bbe-a237-4528983d8d38" volumeName="kubernetes.io/projected/15c3c93f-5487-4bbe-a237-4528983d8d38-kube-api-access-jkcbn" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220795 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34995d02-7971-4439-9d1c-c1083cc72be9" volumeName="kubernetes.io/projected/34995d02-7971-4439-9d1c-c1083cc72be9-kube-api-access-crwwm" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220813 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef87adc-7459-49ed-9810-4482be252165" volumeName="kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220825 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbf066bd-0ff4-471d-ba92-d8cd3d6820fe" volumeName="kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220845 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34995d02-7971-4439-9d1c-c1083cc72be9" volumeName="kubernetes.io/secret/34995d02-7971-4439-9d1c-c1083cc72be9-monitoring-plugin-cert" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220857 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afca2ec8-21f4-4bb6-bc31-009191b47dca" volumeName="kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-tls" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220869 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5b2a72-a248-4a35-b6f3-e13c5f7b015b" volumeName="kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220881 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80ea826e-c81c-4429-9a5d-131a2113fcb3" volumeName="kubernetes.io/configmap/80ea826e-c81c-4429-9a5d-131a2113fcb3-serviceca" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220893 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c154cf87-34c2-411a-bf51-a67f2cd9e1d2" volumeName="kubernetes.io/secret/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-metrics-tls" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220905 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f120a481-1b91-4db7-92c7-eebb3d7e7192" volumeName="kubernetes.io/projected/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-api-access-l82rg" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220917 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbf066bd-0ff4-471d-ba92-d8cd3d6820fe" volumeName="kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-metrics-server-audit-profiles" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220929 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220941 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220954 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15c3c93f-5487-4bbe-a237-4528983d8d38" volumeName="kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220966 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df8f53e-1025-4a88-8945-4055732800c6" volumeName="kubernetes.io/projected/6df8f53e-1025-4a88-8945-4055732800c6-kube-api-access-hf6qp" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220978 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5b2a72-a248-4a35-b6f3-e13c5f7b015b" volumeName="kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-metrics-client-ca" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.220991 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f120a481-1b91-4db7-92c7-eebb3d7e7192" volumeName="kubernetes.io/empty-dir/f120a481-1b91-4db7-92c7-eebb3d7e7192-data-volume" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221003 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa5c470e-0673-4c56-a4e8-feb9fa32b67e" volumeName="kubernetes.io/projected/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-kube-api-access-q9wd9" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221015 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55540c7-35bd-476f-923d-d22e528e3545" volumeName="kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-config" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221028 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221041 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b" volumeName="kubernetes.io/configmap/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-metrics-client-ca" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221053 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef87adc-7459-49ed-9810-4482be252165" volumeName="kubernetes.io/projected/6ef87adc-7459-49ed-9810-4482be252165-kube-api-access-cnxlh" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221065 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221078 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f77eb99-e6b3-4ed7-a9e0-a5667e93807f" volumeName="kubernetes.io/projected/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-kube-api-access-xf7x9" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221089 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7918db7e-5d6b-47bd-bd67-32811f8e40f7" volumeName="kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221102 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80ea826e-c81c-4429-9a5d-131a2113fcb3" volumeName="kubernetes.io/projected/80ea826e-c81c-4429-9a5d-131a2113fcb3-kube-api-access-pkxw4" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221115 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d19fab29-f58d-4668-b635-596a496154ed" volumeName="kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-cni-binary-copy" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221127 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/projected/d9deeffc-4dcb-489c-bbbd-f472272d3c13-kube-api-access-5pjr4" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221138 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e31e6c-2296-4e75-9eed-be8579a717b9" volumeName="kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221150 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f77eb99-e6b3-4ed7-a9e0-a5667e93807f" volumeName="kubernetes.io/configmap/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-iptables-alerter-script" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221207 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221221 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221234 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221246 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221258 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c154cf87-34c2-411a-bf51-a67f2cd9e1d2" volumeName="kubernetes.io/configmap/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-config-volume" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221271 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d19fab29-f58d-4668-b635-596a496154ed" volumeName="kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-multus-daemon-config" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221302 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221316 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221330 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221343 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afca2ec8-21f4-4bb6-bc31-009191b47dca" volumeName="kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221355 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f120a481-1b91-4db7-92c7-eebb3d7e7192" volumeName="kubernetes.io/secret/f120a481-1b91-4db7-92c7-eebb3d7e7192-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221368 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbf066bd-0ff4-471d-ba92-d8cd3d6820fe" volumeName="kubernetes.io/empty-dir/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-audit-log" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221381 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbf066bd-0ff4-471d-ba92-d8cd3d6820fe" volumeName="kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221393 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7918db7e-5d6b-47bd-bd67-32811f8e40f7" volumeName="kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221408 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5b2a72-a248-4a35-b6f3-e13c5f7b015b" volumeName="kubernetes.io/empty-dir/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-volume-directive-shadow" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221420 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e31e6c-2296-4e75-9eed-be8579a717b9" volumeName="kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221433 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e31e6c-2296-4e75-9eed-be8579a717b9" volumeName="kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh" seLinuxMountContext="" Nov 01 00:24:15.221412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221445 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e31e6c-2296-4e75-9eed-be8579a717b9" volumeName="kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221456 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/empty-dir/d9deeffc-4dcb-489c-bbbd-f472272d3c13-tmp" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221470 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5b2a72-a248-4a35-b6f3-e13c5f7b015b" volumeName="kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221486 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221498 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afca2ec8-21f4-4bb6-bc31-009191b47dca" volumeName="kubernetes.io/empty-dir/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-textfile" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221510 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55540c7-35bd-476f-923d-d22e528e3545" volumeName="kubernetes.io/secret/e55540c7-35bd-476f-923d-d22e528e3545-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221522 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f120a481-1b91-4db7-92c7-eebb3d7e7192" volumeName="kubernetes.io/configmap/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221534 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221548 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b" volumeName="kubernetes.io/projected/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-kube-api-access-8x5g6" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221561 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e31e6c-2296-4e75-9eed-be8579a717b9" volumeName="kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221573 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-tls" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221587 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6f778b-df02-4af4-9e02-0d7d40321a52" volumeName="kubernetes.io/secret/de6f778b-df02-4af4-9e02-0d7d40321a52-cert" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221600 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55540c7-35bd-476f-923d-d22e528e3545" volumeName="kubernetes.io/projected/e55540c7-35bd-476f-923d-d22e528e3545-kube-api-access-vqkjh" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221612 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221626 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7918db7e-5d6b-47bd-bd67-32811f8e40f7" volumeName="kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221639 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/configmap/d9deeffc-4dcb-489c-bbbd-f472272d3c13-metrics-client-ca" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221653 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221664 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221676 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ad3371f-f985-4f80-b26f-d69319f24cac" volumeName="kubernetes.io/configmap/7ad3371f-f985-4f80-b26f-d69319f24cac-konnectivity-ca" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221688 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c154cf87-34c2-411a-bf51-a67f2cd9e1d2" volumeName="kubernetes.io/empty-dir/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-tmp-dir" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221699 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221713 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221725 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7918db7e-5d6b-47bd-bd67-32811f8e40f7" volumeName="kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221752 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221765 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6f778b-df02-4af4-9e02-0d7d40321a52" volumeName="kubernetes.io/projected/de6f778b-df02-4af4-9e02-0d7d40321a52-kube-api-access-hvkgw" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221777 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7918db7e-5d6b-47bd-bd67-32811f8e40f7" volumeName="kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221792 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afca2ec8-21f4-4bb6-bc31-009191b47dca" volumeName="kubernetes.io/projected/afca2ec8-21f4-4bb6-bc31-009191b47dca-kube-api-access-pgkh5" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221804 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9deeffc-4dcb-489c-bbbd-f472272d3c13" volumeName="kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221818 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55540c7-35bd-476f-923d-d22e528e3545" volumeName="kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-script-lib" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221832 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa5c470e-0673-4c56-a4e8-feb9fa32b67e" volumeName="kubernetes.io/empty-dir/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-tmp-dir" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221843 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221856 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df8f53e-1025-4a88-8945-4055732800c6" volumeName="kubernetes.io/configmap/6df8f53e-1025-4a88-8945-4055732800c6-metrics-client-ca" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221867 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7918db7e-5d6b-47bd-bd67-32811f8e40f7" volumeName="kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221880 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221892 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5b2a72-a248-4a35-b6f3-e13c5f7b015b" volumeName="kubernetes.io/projected/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-api-access-stflp" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221905 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221916 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221929 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ad3371f-f985-4f80-b26f-d69319f24cac" volumeName="kubernetes.io/secret/7ad3371f-f985-4f80-b26f-d69319f24cac-agent-certs" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221940 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221972 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7918db7e-5d6b-47bd-bd67-32811f8e40f7" volumeName="kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221985 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c154cf87-34c2-411a-bf51-a67f2cd9e1d2" volumeName="kubernetes.io/projected/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-kube-api-access-ljc9d" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.221997 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5b2a72-a248-4a35-b6f3-e13c5f7b015b" volumeName="kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-tls" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222009 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222021 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e31e6c-2296-4e75-9eed-be8579a717b9" volumeName="kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222034 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf719590-ce53-4a44-af67-05c687f68258" volumeName="kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222050 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222062 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df8f53e-1025-4a88-8945-4055732800c6" volumeName="kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222074 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5dd766-39a4-45c1-84e8-fa86b275f100" volumeName="kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-tmp" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222086 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="144d7167-16c0-4f04-a304-ebc22c47283e" volumeName="kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222097 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="186510bd-08db-41ac-9e29-4907eb505d01" volumeName="kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222109 6036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5dd766-39a4-45c1-84e8-fa86b275f100" volumeName="kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-tuned" seLinuxMountContext="" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222122 6036 reconstruct.go:97] "Volume reconstruction finished" Nov 01 00:24:15.223940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.222130 6036 reconciler.go:29] "Reconciler: start to sync state" Nov 01 00:24:15.231639 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.231606 6036 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 00:24:15.242365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.242334 6036 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 00:24:15.242365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.242363 6036 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 00:24:15.242527 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.242382 6036 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 00:24:15.242527 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.242417 6036 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 00:24:15.270634 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.270600 6036 generic.go:356] "Generic (PLEG): container finished" podID="186510bd-08db-41ac-9e29-4907eb505d01" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" exitCode=0 Nov 01 00:24:15.286449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.286412 6036 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="2edc46829be3b0c730b00c503ff3174b03860aaf13c213c93cc5e2d05845bca7" exitCode=0 Nov 01 00:24:15.286449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.286443 6036 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="240bfa78bba9aeede7d28e4a7571f97faac0242bbd458345eefc208a6c510c86" exitCode=0 Nov 01 00:24:15.286449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.286450 6036 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="3862dd8a7a43bb68fe7397f32685c496ed02b8c254a667914d0e620cc1b4ea94" exitCode=0 Nov 01 00:24:15.286449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.286458 6036 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="2ebdd7c83a6afd78534db73663d35e965b481a869b62ad02c511d26def630939" exitCode=0 Nov 01 00:24:15.286741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.286464 6036 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="fc27f404fb62fd39b3a31ddf0264524d4dfb63a8aa10493fc3dcf3c3d419e7f6" exitCode=0 Nov 01 00:24:15.286741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.286469 6036 generic.go:356] "Generic (PLEG): container finished" podID="6ef87adc-7459-49ed-9810-4482be252165" containerID="17889bfe872216eac0538c2d585ba5dbd11ae78812c8741a9f1e4d311cd3bcb3" exitCode=0 Nov 01 00:24:15.300649 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.300616 6036 generic.go:356] "Generic (PLEG): container finished" podID="144d7167-16c0-4f04-a304-ebc22c47283e" containerID="fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169" exitCode=0 Nov 01 00:24:15.308643 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.308613 6036 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-93.us-west-1.compute.internal\" not found" Nov 01 00:24:15.312269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.312230 6036 generic.go:356] "Generic (PLEG): container finished" podID="a74b1dfaae23b56af399f563bf8f80d7" containerID="39a18f4b9b2d0535b7a8eae2a9a4d31ac1a67eb99547f89909fecd77a230645f" exitCode=0 Nov 01 00:24:15.318148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.318108 6036 generic.go:356] "Generic (PLEG): container finished" podID="afca2ec8-21f4-4bb6-bc31-009191b47dca" containerID="fb07ae7f06ab5e064a6056ee9606e94fbafa312130dbe9836cdf0407c14e2cfc" exitCode=0 Nov 01 00:24:15.342798 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.342757 6036 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 00:24:15.345352 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.345332 6036 manager.go:324] Recovery completed Nov 01 00:24:15.369711 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.369681 6036 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:24:15.371898 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.371878 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:24:15.372018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.371915 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:24:15.372018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.371929 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:24:15.375270 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375250 6036 cpu_manager.go:226] "Starting" policy="none" Nov 01 00:24:15.375270 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375264 6036 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 00:24:15.375457 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375265 6036 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:24:15.375457 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375302 6036 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 00:24:15.375457 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375447 6036 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 00:24:15.375552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375460 6036 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 00:24:15.375552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375480 6036 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 00:24:15.375552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375486 6036 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 00:24:15.375552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375493 6036 policy_none.go:50] "Start" Nov 01 00:24:15.375552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375507 6036 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 00:24:15.375552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375516 6036 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 00:24:15.375747 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375654 6036 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 00:24:15.375747 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375661 6036 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 00:24:15.375747 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.375666 6036 policy_none.go:44] "Start" Nov 01 00:24:15.376473 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.376458 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:24:15.376554 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.376487 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:24:15.376554 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.376499 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:24:15.382279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.382246 6036 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 00:24:15.387224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387202 6036 manager.go:341] "Starting Device Plugin manager" Nov 01 00:24:15.387358 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.387237 6036 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 00:24:15.387358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387250 6036 server.go:87] "Starting device plugin registration server" Nov 01 00:24:15.387582 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387568 6036 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 00:24:15.387669 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387582 6036 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 00:24:15.387722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387690 6036 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 00:24:15.387843 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387806 6036 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 00:24:15.387910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387857 6036 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 00:24:15.387910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387863 6036 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 00:24:15.388005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.387916 6036 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T00:24:15.387841437Z","UUID":"6e45a18f-1fa7-4709-ac1f-414d6f8174ca","Handler":null,"Name":"","Endpoint":""} Nov 01 00:24:15.389596 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.389573 6036 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 00:24:15.389699 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.389618 6036 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-93.us-west-1.compute.internal\" not found" Nov 01 00:24:15.390166 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.390149 6036 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 00:24:15.390250 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.390171 6036 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 00:24:15.407256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.407227 6036 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:24:15.407424 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.407227 6036 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:24:15.407525 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.407470 6036 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 00:24:15.407893 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.407879 6036 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:24:15.487751 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.487713 6036 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:24:15.489062 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.489042 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:24:15.489128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.489078 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:24:15.489128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.489092 6036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-93.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:24:15.489182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.489143 6036 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.507994 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.507965 6036 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:24:15.543784 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.543682 6036 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal"] Nov 01 00:24:15.544536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.544513 6036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.544536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.544502 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"a74b1dfaae23b56af399f563bf8f80d7","Type":"ContainerStarted","Data":"dd19dc94f53f43d860fb31b855e6b322530f363909e38ee273a4a9b624c89faf"} Nov 01 00:24:15.544665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.544557 6036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.544665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.544560 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"a74b1dfaae23b56af399f563bf8f80d7","Type":"ContainerDied","Data":"39a18f4b9b2d0535b7a8eae2a9a4d31ac1a67eb99547f89909fecd77a230645f"} Nov 01 00:24:15.544665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.544582 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"a74b1dfaae23b56af399f563bf8f80d7","Type":"ContainerStarted","Data":"67d02bfe87504573f9473b10cb83becef096897d9004e7553b8145d0934b5bdb"} Nov 01 00:24:15.544665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.544599 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"57ff4fc49ea0c4ad7e09012148b2302a","Type":"ContainerStarted","Data":"5f551eca9a3d02b351bd948310aadb47a9f285ac77db5e1ced21ecd9f4a34dfb"} Nov 01 00:24:15.544665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.544616 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" event={"ID":"57ff4fc49ea0c4ad7e09012148b2302a","Type":"ContainerStarted","Data":"d568f7d494cf16f683795b12308fa36422f85d82cd5de081c8ae4cdfa2139186"} Nov 01 00:24:15.607862 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.607832 6036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:24:15.608004 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.607884 6036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.608004 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.607984 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.608071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.608002 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.608071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.608021 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ff4fc49ea0c4ad7e09012148b2302a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"57ff4fc49ea0c4ad7e09012148b2302a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.608465 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.608448 6036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:24:15.608508 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:15.608490 6036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.619130 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.619107 6036 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.619191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.619181 6036 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.708895 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.708855 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.708895 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.708892 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.709080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.708911 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ff4fc49ea0c4ad7e09012148b2302a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"57ff4fc49ea0c4ad7e09012148b2302a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.709080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.708940 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.709080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.708969 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a74b1dfaae23b56af399f563bf8f80d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"a74b1dfaae23b56af399f563bf8f80d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:15.709080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:15.708949 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ff4fc49ea0c4ad7e09012148b2302a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal\" (UID: \"57ff4fc49ea0c4ad7e09012148b2302a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:16.192035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.192001 6036 apiserver.go:52] "Watching apiserver" Nov 01 00:24:16.300669 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.300622 6036 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 00:24:16.301445 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.301416 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-546d87964-scgxb","kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq","openshift-console/console-585bf7664f-h64rc","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz","openshift-insights/insights-runtime-extractor-9rxd6","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-5d695","openshift-dns/dns-default-wt9s4","openshift-dns/node-resolver-f994g","openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf","openshift-monitoring/metrics-server-5db84b6cc6-rp9xt","kube-system/global-pull-secret-syncer-x4r6x","openshift-multus/multus-j5dgf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-d74nd","openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn","openshift-cluster-node-tuning-operator/tuned-hpcm2","openshift-ingress-canary/ingress-canary-pz786","openshift-multus/multus-additional-cni-plugins-w9d7m","kube-system/konnectivity-agent-2j99d","openshift-image-registry/node-ca-qvsc4","openshift-monitoring/monitoring-plugin-77f4f76594-d85jb","openshift-network-diagnostics/network-check-target-9nfbp","openshift-multus/network-metrics-daemon-v2mlf","openshift-ovn-kubernetes/ovnkube-node-q2d7g","openshift-console/console-5665b49b4d-hvmbv"] Nov 01 00:24:16.311663 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311633 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-cni-binary-copy\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.311663 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311667 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-k8s-cni-cncf-io\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.311910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311683 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-cnibin\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.311910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311698 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-slash\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.311910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311720 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.311910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311757 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-bin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.311910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311832 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.311910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311873 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311927 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-host\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.311958 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312018 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312044 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3371f-f985-4f80-b26f-d69319f24cac-konnectivity-ca\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312069 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zvl\" (UniqueName: \"kubernetes.io/projected/3e567114-6e47-4337-9776-1ec7f47be0b8-kube-api-access-29zvl\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312094 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-netd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312117 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ad3371f-f985-4f80-b26f-d69319f24cac-agent-certs\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312142 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312164 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-run\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.312205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312193 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-sys\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312215 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-var-lib-kubelet\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312238 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-tuned\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312268 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-socket-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312313 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-system-cni-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312408 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-etc-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312446 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55540c7-35bd-476f-923d-d22e528e3545-ovn-node-metrics-cert\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312447 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-tuned\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312487 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-netns\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312563 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f120a481-1b91-4db7-92c7-eebb3d7e7192-data-volume\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.312626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312598 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxlh\" (UniqueName: \"kubernetes.io/projected/6ef87adc-7459-49ed-9810-4482be252165-kube-api-access-cnxlh\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312659 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-sys-fs\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312682 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ea826e-c81c-4429-9a5d-131a2113fcb3-host\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312688 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f120a481-1b91-4db7-92c7-eebb3d7e7192-data-volume\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312737 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312778 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-dbus\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312812 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-cnibin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312842 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-env-overrides\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312894 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312935 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-kubelet\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312972 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsz6b\" (UniqueName: \"kubernetes.io/projected/d19fab29-f58d-4668-b635-596a496154ed-kube-api-access-tsz6b\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.312999 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-modprobe-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313025 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcbn\" (UniqueName: \"kubernetes.io/projected/15c3c93f-5487-4bbe-a237-4528983d8d38-kube-api-access-jkcbn\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:24:16.313061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313051 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80ea826e-c81c-4429-9a5d-131a2113fcb3-serviceca\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313078 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxw4\" (UniqueName: \"kubernetes.io/projected/80ea826e-c81c-4429-9a5d-131a2113fcb3-kube-api-access-pkxw4\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313103 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-kubelet\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313129 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-var-lib-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313236 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-etc-kubernetes\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313265 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313324 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313385 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-log-socket\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313418 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-hostroot\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313464 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-host-slash\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313505 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f120a481-1b91-4db7-92c7-eebb3d7e7192-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313532 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313574 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-tmp-dir\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313627 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-multus-daemon-config\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.313677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313674 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313687 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-tmp-dir\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313758 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-systemd\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313809 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9wd9\" (UniqueName: \"kubernetes.io/projected/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-kube-api-access-q9wd9\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.313888 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-conf\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314021 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-lib-modules\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314087 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314122 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-script-lib\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314156 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-iptables-alerter-script\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314189 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7x9\" (UniqueName: \"kubernetes.io/projected/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-kube-api-access-xf7x9\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314224 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysconfig\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.314278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314255 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjmz\" (UniqueName: \"kubernetes.io/projected/bd5dd766-39a4-45c1-84e8-fa86b275f100-kube-api-access-9wjmz\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314307 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314336 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-systemd-units\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314359 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-bin\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314393 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-kubelet-config\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314439 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-tmp\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314461 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82rg\" (UniqueName: \"kubernetes.io/projected/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-api-access-l82rg\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314500 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-node-log\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314617 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314634 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5dd766-39a4-45c1-84e8-fa86b275f100-tmp\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314669 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-system-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314705 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-netns\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.314783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314753 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-systemd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.315308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314795 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-config\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.315308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314844 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-socket-dir-parent\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.315308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314903 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-conf-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.315308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314940 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f120a481-1b91-4db7-92c7-eebb3d7e7192-crio-socket\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.315308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.314974 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-registration-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.315702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.315673 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-hosts-file\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:24:16.315785 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.315745 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-ovn\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.315849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.315829 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkjh\" (UniqueName: \"kubernetes.io/projected/e55540c7-35bd-476f-923d-d22e528e3545-kube-api-access-vqkjh\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.316068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.316034 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-multus\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.316155 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.316102 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-multus-certs\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.316155 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.316126 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-kubernetes\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.316248 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.316170 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-device-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.326041 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.326015 6036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:16.326180 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.326115 6036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:16.355798 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.355770 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 00:24:16.356214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356195 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 00:24:16.356260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356246 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m6gx8\"" Nov 01 00:24:16.356466 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356455 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.356574 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356556 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.356624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356603 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 00:24:16.356624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356560 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 00:24:16.356624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356611 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rjfbh\"" Nov 01 00:24:16.356735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356562 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nz8nc\"" Nov 01 00:24:16.356735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356682 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 00:24:16.356735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356687 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 00:24:16.356735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356685 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zctdk\"" Nov 01 00:24:16.356886 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356774 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.356886 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356818 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.356886 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356823 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 00:24:16.356983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356898 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 00:24:16.356983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356914 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rhkgk\"" Nov 01 00:24:16.356983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.356979 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 00:24:16.357087 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357024 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.357182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357166 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.357585 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357565 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6c54w\"" Nov 01 00:24:16.357657 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357610 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 00:24:16.357657 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357626 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.357762 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357692 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 00:24:16.357762 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357702 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 00:24:16.357762 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357758 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 00:24:16.357891 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357806 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.357891 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357826 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.357891 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357886 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.358038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357888 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.358038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.357951 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cx8lz\"" Nov 01 00:24:16.358038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358007 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k669s\"" Nov 01 00:24:16.358038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358015 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.358178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358041 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.358263 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358241 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4lrql\"" Nov 01 00:24:16.358344 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358263 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t74k4\"" Nov 01 00:24:16.358344 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358303 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.358344 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358313 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.358621 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358604 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hhlh7\"" Nov 01 00:24:16.358621 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358617 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.358722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358647 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 00:24:16.358823 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358810 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 00:24:16.358855 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358848 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7ckpz\"" Nov 01 00:24:16.358989 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.358974 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqdjj\"" Nov 01 00:24:16.362233 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.362217 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-cni-binary-copy\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.362420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.362403 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c3c93f-5487-4bbe-a237-4528983d8d38-metrics-certs\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:24:16.362499 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.362454 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.362558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.362525 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3371f-f985-4f80-b26f-d69319f24cac-konnectivity-ca\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:24:16.362605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.362554 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ad3371f-f985-4f80-b26f-d69319f24cac-agent-certs\") pod \"konnectivity-agent-2j99d\" (UID: \"7ad3371f-f985-4f80-b26f-d69319f24cac\") " pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:24:16.362745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.362726 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55540c7-35bd-476f-923d-d22e528e3545-ovn-node-metrics-cert\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.362745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.362739 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f953789-9f33-475b-9687-714c6960973c-original-pull-secret\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:16.364388 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.364369 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-env-overrides\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.364476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.364370 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.364476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.364384 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d19fab29-f58d-4668-b635-596a496154ed-multus-daemon-config\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.364476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.364405 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-script-lib\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.364620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.364492 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-iptables-alerter-script\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:24:16.364620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.364581 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ef87adc-7459-49ed-9810-4482be252165-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.364831 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.364810 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80ea826e-c81c-4429-9a5d-131a2113fcb3-serviceca\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:24:16.365602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.365587 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55540c7-35bd-476f-923d-d22e528e3545-ovnkube-config\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.400682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.400623 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.400682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.400636 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 00:24:16.400682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.400675 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.400682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.400683 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.400682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.400636 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5tc5v\"" Nov 01 00:24:16.401006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.400701 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.401006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.400745 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.401067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.401048 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 00:24:16.401067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.401049 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 00:24:16.401147 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.401051 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 00:24:16.402089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402064 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 00:24:16.402200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402099 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 00:24:16.402237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402226 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 00:24:16.402553 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402535 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dqtrs\"" Nov 01 00:24:16.402634 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402622 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 00:24:16.402686 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402642 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 00:24:16.402750 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402739 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 00:24:16.402801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402771 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 00:24:16.402835 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.402806 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwm8b\"" Nov 01 00:24:16.403054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.403039 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.404171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.404152 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f120a481-1b91-4db7-92c7-eebb3d7e7192-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.406528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.406514 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vqqjw\"" Nov 01 00:24:16.408311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.408295 6036 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 00:24:16.410637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.410617 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsz6b\" (UniqueName: \"kubernetes.io/projected/d19fab29-f58d-4668-b635-596a496154ed-kube-api-access-tsz6b\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.410637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.410633 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7x9\" (UniqueName: \"kubernetes.io/projected/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-kube-api-access-xf7x9\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:24:16.410773 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.410753 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zvl\" (UniqueName: \"kubernetes.io/projected/3e567114-6e47-4337-9776-1ec7f47be0b8-kube-api-access-29zvl\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.410808 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.410783 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxlh\" (UniqueName: \"kubernetes.io/projected/6ef87adc-7459-49ed-9810-4482be252165-kube-api-access-cnxlh\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.411083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.411069 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxw4\" (UniqueName: \"kubernetes.io/projected/80ea826e-c81c-4429-9a5d-131a2113fcb3-kube-api-access-pkxw4\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:24:16.411237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.411222 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495mv\" (UniqueName: \"kubernetes.io/projected/bf719590-ce53-4a44-af67-05c687f68258-kube-api-access-495mv\") pod \"network-check-target-9nfbp\" (UID: \"bf719590-ce53-4a44-af67-05c687f68258\") " pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:24:16.416597 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416575 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-k8s-cni-cncf-io\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.416644 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416614 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-tmp-dir\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:16.416696 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416675 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-k8s-cni-cncf-io\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.416745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416701 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-tmp-dir\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:16.416745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416727 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-sys\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.416811 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416755 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.416811 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416775 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.416811 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416792 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-slash\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.416946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416827 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.416946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416854 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-bin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.416946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416877 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-host\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.416946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416893 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-slash\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.416946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416903 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:24:16.416946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416934 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.416946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416945 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-bin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416965 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416984 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.416993 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417020 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417044 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417068 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417101 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417136 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9g6\" (UniqueName: \"kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417167 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417176 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82rg\" (UniqueName: \"kubernetes.io/projected/f120a481-1b91-4db7-92c7-eebb3d7e7192-kube-api-access-l82rg\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417192 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-netd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417179 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9wd9\" (UniqueName: \"kubernetes.io/projected/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-kube-api-access-q9wd9\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417185 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417237 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-host\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417246 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-sys\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.417279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417270 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-var-lib-kubelet\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417313 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjmz\" (UniqueName: \"kubernetes.io/projected/bd5dd766-39a4-45c1-84e8-fa86b275f100-kube-api-access-9wjmz\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417317 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-socket-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417323 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417344 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ea826e-c81c-4429-9a5d-131a2113fcb3-host\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417368 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417383 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417390 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-netd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417392 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-etc-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417431 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljc9d\" (UniqueName: \"kubernetes.io/projected/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-kube-api-access-ljc9d\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417442 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-socket-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417442 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-etc-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417460 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2q7j\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417510 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkjh\" (UniqueName: \"kubernetes.io/projected/e55540c7-35bd-476f-923d-d22e528e3545-kube-api-access-vqkjh\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417501 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-var-lib-kubelet\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417526 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ea826e-c81c-4429-9a5d-131a2113fcb3-host\") pod \"node-ca-qvsc4\" (UID: \"80ea826e-c81c-4429-9a5d-131a2113fcb3\") " pod="openshift-image-registry/node-ca-qvsc4" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417515 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417507 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de6f778b-df02-4af4-9e02-0d7d40321a52-cert\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417550 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcbn\" (UniqueName: \"kubernetes.io/projected/15c3c93f-5487-4bbe-a237-4528983d8d38-kube-api-access-jkcbn\") pod \"network-metrics-daemon-v2mlf\" (UID: \"15c3c93f-5487-4bbe-a237-4528983d8d38\") " pod="openshift-multus/network-metrics-daemon-v2mlf" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417554 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-sys\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417660 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417681 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417694 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-wtmp\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417722 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417754 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stflp\" (UniqueName: \"kubernetes.io/projected/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-api-access-stflp\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417839 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-dbus\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417863 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417878 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417900 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ktk\" (UniqueName: \"kubernetes.io/projected/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-kube-api-access-t8ktk\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417917 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-modprobe-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417927 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de6f778b-df02-4af4-9e02-0d7d40321a52-cert\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417939 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417949 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417972 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417995 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-kubelet\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.418002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418017 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-var-lib-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418042 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418062 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-modprobe-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418067 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6qp\" (UniqueName: \"kubernetes.io/projected/6df8f53e-1025-4a88-8945-4055732800c6-kube-api-access-hf6qp\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418069 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-var-lib-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.417982 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-dbus\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418086 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418097 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxjh\" (UniqueName: \"kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418127 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418126 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-kubelet\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418144 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418185 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-d\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418189 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418257 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-host-slash\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418365 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f77eb99-e6b3-4ed7-a9e0-a5667e93807f-host-slash\") pod \"iptables-alerter-d74nd\" (UID: \"3f77eb99-e6b3-4ed7-a9e0-a5667e93807f\") " pod="openshift-network-operator/iptables-alerter-d74nd" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418387 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418392 6036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418405 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:16.418434 6036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418449 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418466 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-grpc-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418504 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-client-certs\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418530 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418545 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-metrics-client-ca\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418560 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-conf\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418575 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-lib-modules\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418597 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418667 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418695 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-lib-modules\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418708 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysctl-conf\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418737 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418766 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-metrics-client-ca\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418768 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418859 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418873 6036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418889 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:16.418916 6036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.us-west-1.compute.internal" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418922 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418950 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysconfig\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.418988 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419011 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-openvswitch\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419018 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419023 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419041 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419031 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-sysconfig\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419074 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-bin\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419101 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-tls\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419128 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-node-log\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419136 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-cni-bin\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419155 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419175 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419183 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-system-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419184 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-node-log\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419212 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-metrics-tls\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419226 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419238 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419224 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419246 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-system-cni-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419272 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419326 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419352 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-systemd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419376 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-registration-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419398 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-metrics-tls\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419401 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419420 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-systemd\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419446 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419469 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-multus\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.419432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419446 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-registration-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419492 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-multus-certs\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419510 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-kubernetes\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419514 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-cni-multus\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419529 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-device-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419548 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-root\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419566 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9deeffc-4dcb-489c-bbbd-f472272d3c13-metrics-client-ca\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419593 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-kubernetes\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419605 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-multus-certs\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419612 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-device-dir\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419613 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwwm\" (UniqueName: \"kubernetes.io/projected/34995d02-7971-4439-9d1c-c1083cc72be9-kube-api-access-crwwm\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419647 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-cnibin\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419664 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419679 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-client-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419690 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-cnibin\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419695 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419713 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419728 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-config-volume\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419753 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419778 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-textfile\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419813 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419829 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9deeffc-4dcb-489c-bbbd-f472272d3c13-metrics-client-ca\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419869 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419873 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-textfile\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419890 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419908 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-config-volume\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419923 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.419944 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420000 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-run\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420034 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-sys-fs\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420062 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420064 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-run\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420120 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-system-cni-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420135 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-sys-fs\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420151 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-netns\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420187 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420215 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-run-netns\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420213 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9deeffc-4dcb-489c-bbbd-f472272d3c13-tmp\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420191 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef87adc-7459-49ed-9810-4482be252165-system-cni-dir\") pod \"multus-additional-cni-plugins-w9d7m\" (UID: \"6ef87adc-7459-49ed-9810-4482be252165\") " pod="openshift-multus/multus-additional-cni-plugins-w9d7m" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420255 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjr4\" (UniqueName: \"kubernetes.io/projected/d9deeffc-4dcb-489c-bbbd-f472272d3c13-kube-api-access-5pjr4\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420278 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420279 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9deeffc-4dcb-489c-bbbd-f472272d3c13-tmp\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420318 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-cnibin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420343 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-cnibin\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420363 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-kubelet\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420378 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-host-var-lib-kubelet\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420394 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-kube-api-access-8x5g6\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420431 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-accelerators-collector-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420459 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420489 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420515 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420544 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g85t\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420570 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-etc-kubernetes\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420597 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420601 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-etc-kubernetes\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420623 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420651 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkgw\" (UniqueName: \"kubernetes.io/projected/de6f778b-df02-4af4-9e02-0d7d40321a52-kube-api-access-hvkgw\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420678 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420704 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-log-socket\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420729 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-hostroot\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420738 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e567114-6e47-4337-9776-1ec7f47be0b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-fk2dq\" (UID: \"3e567114-6e47-4337-9776-1ec7f47be0b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fk2dq" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420758 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420761 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-log-socket\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420808 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-hostroot\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420850 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-systemd\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420871 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420887 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420914 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5dd766-39a4-45c1-84e8-fa86b275f100-etc-systemd\") pod \"tuned-hpcm2\" (UID: \"bd5dd766-39a4-45c1-84e8-fa86b275f100\") " pod="openshift-cluster-node-tuning-operator/tuned-hpcm2" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420914 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-systemd-units\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420939 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-kubelet-config\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420966 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-systemd-units\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420970 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420991 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f953789-9f33-475b-9687-714c6960973c-kubelet-config\") pod \"global-pull-secret-syncer-x4r6x\" (UID: \"5f953789-9f33-475b-9687-714c6960973c\") " pod="kube-system/global-pull-secret-syncer-x4r6x" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.420992 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421002 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6df8f53e-1025-4a88-8945-4055732800c6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421023 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421046 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34995d02-7971-4439-9d1c-c1083cc72be9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421063 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-audit-log\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421078 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421094 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-tls\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421099 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421113 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421139 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421151 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-audit-log\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421166 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-netns\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421175 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6df8f53e-1025-4a88-8945-4055732800c6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421190 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-socket-dir-parent\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421207 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-conf-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421222 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f120a481-1b91-4db7-92c7-eebb3d7e7192-crio-socket\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421190 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421245 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-hosts-file\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421265 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-socket-dir-parent\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421272 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421275 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d19fab29-f58d-4668-b635-596a496154ed-multus-conf-dir\") pod \"multus-j5dgf\" (UID: \"d19fab29-f58d-4668-b635-596a496154ed\") " pod="openshift-multus/multus-j5dgf" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421297 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-host-run-netns\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421315 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421333 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f120a481-1b91-4db7-92c7-eebb3d7e7192-crio-socket\") pod \"insights-runtime-extractor-9rxd6\" (UID: \"f120a481-1b91-4db7-92c7-eebb3d7e7192\") " pod="openshift-insights/insights-runtime-extractor-9rxd6" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421341 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-ovn\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421348 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa5c470e-0673-4c56-a4e8-feb9fa32b67e-hosts-file\") pod \"node-resolver-f994g\" (UID: \"fa5c470e-0673-4c56-a4e8-feb9fa32b67e\") " pod="openshift-dns/node-resolver-f994g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421374 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421386 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55540c7-35bd-476f-923d-d22e528e3545-run-ovn\") pod \"ovnkube-node-q2d7g\" (UID: \"e55540c7-35bd-476f-923d-d22e528e3545\") " pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421404 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421427 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkh5\" (UniqueName: \"kubernetes.io/projected/afca2ec8-21f4-4bb6-bc31-009191b47dca-kube-api-access-pgkh5\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421444 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-metrics-server-audit-profiles\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421464 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.421949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.421518 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:24:16.455938 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.455848 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 00:24:16.456089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.455998 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 00:24:16.456089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456024 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 00:24:16.456089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456035 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.456089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456077 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 00:24:16.456321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456098 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 00:24:16.456321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456041 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 00:24:16.456321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456233 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 00:24:16.456321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456229 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 00:24:16.456490 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456424 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 00:24:16.456552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456491 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 00:24:16.456745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456728 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 00:24:16.456811 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456781 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 00:24:16.457032 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.456970 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457125 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457142 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457142 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457188 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457155 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457157 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457125 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457251 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457267 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457276 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457313 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457379 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457384 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457475 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-75fpm3fqiecoh\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457484 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457502 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-b2d4t\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457505 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457528 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457550 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457564 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457621 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457628 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457633 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457660 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457684 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457700 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7rnajk1nfvaac\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457736 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 00:24:16.457725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457761 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457765 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457806 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vngfn\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457844 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457866 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d9w4t\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457905 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d652d\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457920 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457950 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.457951 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wfz2h\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458055 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458094 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458127 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8rsns\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458171 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j7jdw\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458229 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458246 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xl8bk\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458318 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458476 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458602 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.458894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.458607 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459127 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459181 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-client-certs\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459190 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-grpc-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459269 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459366 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459392 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459517 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-936to0tetgr6n\"" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459559 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-tls\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459572 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459650 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.459741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459693 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.460205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459785 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.460205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.459939 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.460205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460104 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.460356 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460275 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.460530 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460512 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-client-ca-bundle\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.460581 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460567 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.460690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460673 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.460811 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460791 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.460870 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460823 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.460965 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460947 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.461018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460972 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.461018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460994 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.461128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.460997 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.461182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461126 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.461182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461142 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.461182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461156 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.461413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461218 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6df8f53e-1025-4a88-8945-4055732800c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:24:16.461413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461326 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.461413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461367 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-accelerators-collector-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.461413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461380 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.461621 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461489 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.461621 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461533 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.461621 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461594 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-secret-metrics-server-tls\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.461776 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461533 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34995d02-7971-4439-9d1c-c1083cc72be9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:16.461776 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461681 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.461776 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461703 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.461776 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461728 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-tls\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.461949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461825 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-metrics-server-audit-profiles\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.461998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.461957 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.467194 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.467173 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9deeffc-4dcb-489c-bbbd-f472272d3c13-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.467501 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.467483 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.467607 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.467594 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.507931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.507895 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.508435 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.508762 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljc9d\" (UniqueName: \"kubernetes.io/projected/c154cf87-34c2-411a-bf51-a67f2cd9e1d2-kube-api-access-ljc9d\") pod \"dns-default-wt9s4\" (UID: \"c154cf87-34c2-411a-bf51-a67f2cd9e1d2\") " pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.508811 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.508952 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stflp\" (UniqueName: \"kubernetes.io/projected/dc5b2a72-a248-4a35-b6f3-e13c5f7b015b-kube-api-access-stflp\") pod \"kube-state-metrics-5dc8d7d89b-qc5gf\" (UID: \"dc5b2a72-a248-4a35-b6f3-e13c5f7b015b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qc5gf" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509027 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9g6\" (UniqueName: \"kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6\") pod \"console-5665b49b4d-hvmbv\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509079 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ktk\" (UniqueName: \"kubernetes.io/projected/fbf066bd-0ff4-471d-ba92-d8cd3d6820fe-kube-api-access-t8ktk\") pod \"metrics-server-5db84b6cc6-rp9xt\" (UID: \"fbf066bd-0ff4-471d-ba92-d8cd3d6820fe\") " pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509079 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2q7j\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509238 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwwm\" (UniqueName: \"kubernetes.io/projected/34995d02-7971-4439-9d1c-c1083cc72be9-kube-api-access-crwwm\") pod \"monitoring-plugin-77f4f76594-d85jb\" (UID: \"34995d02-7971-4439-9d1c-c1083cc72be9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509491 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkh5\" (UniqueName: \"kubernetes.io/projected/afca2ec8-21f4-4bb6-bc31-009191b47dca-kube-api-access-pgkh5\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509555 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509787 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxjh\" (UniqueName: \"kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509775 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkgw\" (UniqueName: \"kubernetes.io/projected/de6f778b-df02-4af4-9e02-0d7d40321a52-kube-api-access-hvkgw\") pod \"ingress-canary-pz786\" (UID: \"de6f778b-df02-4af4-9e02-0d7d40321a52\") " pod="openshift-ingress-canary/ingress-canary-pz786" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509828 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b-kube-api-access-8x5g6\") pod \"prometheus-operator-9cd89b74c-6fxrz\" (UID: \"24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6fxrz" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509846 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjr4\" (UniqueName: \"kubernetes.io/projected/d9deeffc-4dcb-489c-bbbd-f472272d3c13-kube-api-access-5pjr4\") pod \"thanos-querier-546d87964-scgxb\" (UID: \"d9deeffc-4dcb-489c-bbbd-f472272d3c13\") " pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509851 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle\") pod \"console-585bf7664f-h64rc\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:16.509894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509830 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6qp\" (UniqueName: \"kubernetes.io/projected/6df8f53e-1025-4a88-8945-4055732800c6-kube-api-access-hf6qp\") pod \"openshift-state-metrics-7bd7489957-5grgn\" (UID: \"6df8f53e-1025-4a88-8945-4055732800c6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5grgn" Nov 01 00:24:16.510622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.509943 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g85t\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.510622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.510389 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.512277 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.512259 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:16.517869 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.517842 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:16.522865 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.522846 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-root\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.522917 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.522896 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-sys\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.522948 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.522924 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-wtmp\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.523009 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.522964 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-root\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.523009 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.522979 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-sys\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:16.523065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:16.523003 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afca2ec8-21f4-4bb6-bc31-009191b47dca-node-exporter-wtmp\") pod \"node-exporter-5d695\" (UID: \"afca2ec8-21f4-4bb6-bc31-009191b47dca\") " pod="openshift-monitoring/node-exporter-5d695" Nov 01 00:24:17.304534 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:17.304505 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d85jb" Nov 01 00:24:17.998476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:17.998443 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wt9s4" Nov 01 00:24:18.302916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:18.302882 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-546d87964-scgxb" Nov 01 00:24:18.344904 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:18.344878 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:24:19.511572 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:19.511537 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:24:20.166646 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:20.166610 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:20.394112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:20.394078 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:20.854031 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:20.853990 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:20.870617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:20.870588 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:21.822191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:21.822162 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2j99d" Nov 01 00:24:22.139826 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:22.139744 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:24:24.301202 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:24.301127 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:25.788489 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:25.788461 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:25.788913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:25.788657 6036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 00:24:26.927946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:26.927914 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:27.138397 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:27.138359 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:31.651205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:31.651162 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q2d7g" Nov 01 00:24:32.786971 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:32.786942 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:32.992236 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:32.992202 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:41.332277 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.332219 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5665b49b4d-hvmbv" podUID="7918db7e-5d6b-47bd-bd67-32811f8e40f7" containerName="console" containerID="cri-o://549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a" gracePeriod=15 Nov 01 00:24:41.592150 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.592087 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:41.762827 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.762785 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca\") pod \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " Nov 01 00:24:41.763020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.762839 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6\" (UniqueName: \"kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6\") pod \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " Nov 01 00:24:41.763020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.762935 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert\") pod \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " Nov 01 00:24:41.763020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763005 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp\") pod \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " Nov 01 00:24:41.763197 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763031 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert\") pod \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " Nov 01 00:24:41.763197 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763077 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config\") pod \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " Nov 01 00:24:41.763197 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763105 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config\") pod \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\" (UID: \"7918db7e-5d6b-47bd-bd67-32811f8e40f7\") " Nov 01 00:24:41.763382 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763345 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp" pod "7918db7e-5d6b-47bd-bd67-32811f8e40f7" (UID: "7918db7e-5d6b-47bd-bd67-32811f8e40f7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:24:41.763440 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763376 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca" pod "7918db7e-5d6b-47bd-bd67-32811f8e40f7" (UID: "7918db7e-5d6b-47bd-bd67-32811f8e40f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:41.763505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763488 6036 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:41.763541 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763492 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert" pod "7918db7e-5d6b-47bd-bd67-32811f8e40f7" (UID: "7918db7e-5d6b-47bd-bd67-32811f8e40f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:41.763541 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763512 6036 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7918db7e-5d6b-47bd-bd67-32811f8e40f7-tmp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:41.763541 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.763520 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config" pod "7918db7e-5d6b-47bd-bd67-32811f8e40f7" (UID: "7918db7e-5d6b-47bd-bd67-32811f8e40f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:41.765478 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.765445 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert" pod "7918db7e-5d6b-47bd-bd67-32811f8e40f7" (UID: "7918db7e-5d6b-47bd-bd67-32811f8e40f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:41.765603 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.765519 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6" pod "7918db7e-5d6b-47bd-bd67-32811f8e40f7" (UID: "7918db7e-5d6b-47bd-bd67-32811f8e40f7"). InnerVolumeSpecName "kube-api-access-bq9g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:24:41.765603 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.765534 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config" pod "7918db7e-5d6b-47bd-bd67-32811f8e40f7" (UID: "7918db7e-5d6b-47bd-bd67-32811f8e40f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:41.864957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.864853 6036 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:41.864957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.864903 6036 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-oauth-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:41.864957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.864917 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9g6\" (UniqueName: \"kubernetes.io/projected/7918db7e-5d6b-47bd-bd67-32811f8e40f7-kube-api-access-bq9g6\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:41.864957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.864928 6036 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7918db7e-5d6b-47bd-bd67-32811f8e40f7-console-serving-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:41.864957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.864957 6036 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7918db7e-5d6b-47bd-bd67-32811f8e40f7-oauth-serving-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:41.949084 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:41.949051 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:42.193492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.193413 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:42.440358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.440323 6036 generic.go:356] "Generic (PLEG): container finished" podID="7918db7e-5d6b-47bd-bd67-32811f8e40f7" containerID="549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a" exitCode=0 Nov 01 00:24:42.440787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.440368 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5665b49b4d-hvmbv" Nov 01 00:24:42.440787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.440388 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5665b49b4d-hvmbv" event={"ID":"7918db7e-5d6b-47bd-bd67-32811f8e40f7","Type":"ContainerDied","Data":"549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a"} Nov 01 00:24:42.440787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.440437 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5665b49b4d-hvmbv" event={"ID":"7918db7e-5d6b-47bd-bd67-32811f8e40f7","Type":"ContainerDied","Data":"ce2a983f542730942e817f2aab8cc51ef2c45b446a1dbd19ab7a3a7ab511c7c8"} Nov 01 00:24:42.440787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.440458 6036 scope.go:122] "RemoveContainer" containerID="549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a" Nov 01 00:24:42.449380 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.449360 6036 scope.go:122] "RemoveContainer" containerID="549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a" Nov 01 00:24:42.449642 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:42.449623 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a\": container with ID starting with 549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a not found: ID does not exist" containerID="549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a" Nov 01 00:24:42.449712 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.449653 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a"} err="failed to get container status \"549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a\": rpc error: code = NotFound desc = could not find container \"549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a\": container with ID starting with 549d94ca5c50a44ea41978f9012c8c0be565865c8b6a4cce9f221d5a5845251a not found: ID does not exist" Nov 01 00:24:42.594686 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.594652 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:42.606130 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.606100 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5665b49b4d-hvmbv"] Nov 01 00:24:42.656597 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.656562 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5665b49b4d-hvmbv"] Nov 01 00:24:42.804199 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:42.804157 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:43.247690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:43.247617 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7918db7e-5d6b-47bd-bd67-32811f8e40f7" path="/var/lib/kubelet/pods/7918db7e-5d6b-47bd-bd67-32811f8e40f7/volumes" Nov 01 00:24:44.330749 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:44.330720 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5db84b6cc6-rp9xt" Nov 01 00:24:49.504682 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:49.504642 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:24:50.800144 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:50.800106 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:51.060605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:51.060503 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:52.142101 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:52.142073 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9nfbp" Nov 01 00:24:52.928452 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:52.928418 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:24:52.928730 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:52.928707 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="alertmanager" containerID="cri-o://031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010" gracePeriod=120 Nov 01 00:24:52.928820 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:52.928788 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy-metric" containerID="cri-o://8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d" gracePeriod=120 Nov 01 00:24:52.928879 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:52.928836 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="config-reloader" containerID="cri-o://087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1" gracePeriod=120 Nov 01 00:24:52.928936 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:52.928870 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="prom-label-proxy" containerID="cri-o://9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4" gracePeriod=120 Nov 01 00:24:52.928993 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:52.928917 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy" containerID="cri-o://19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72" gracePeriod=120 Nov 01 00:24:52.929708 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:52.928808 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy-web" containerID="cri-o://7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b" gracePeriod=120 Nov 01 00:24:53.493508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.493476 6036 generic.go:356] "Generic (PLEG): container finished" podID="144d7167-16c0-4f04-a304-ebc22c47283e" containerID="9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4" exitCode=0 Nov 01 00:24:53.493508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.493499 6036 generic.go:356] "Generic (PLEG): container finished" podID="144d7167-16c0-4f04-a304-ebc22c47283e" containerID="19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72" exitCode=0 Nov 01 00:24:53.493508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.493505 6036 generic.go:356] "Generic (PLEG): container finished" podID="144d7167-16c0-4f04-a304-ebc22c47283e" containerID="087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1" exitCode=0 Nov 01 00:24:53.493508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.493511 6036 generic.go:356] "Generic (PLEG): container finished" podID="144d7167-16c0-4f04-a304-ebc22c47283e" containerID="031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010" exitCode=0 Nov 01 00:24:53.493985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.493558 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerDied","Data":"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4"} Nov 01 00:24:53.493985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.493629 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerDied","Data":"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72"} Nov 01 00:24:53.493985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.493656 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerDied","Data":"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1"} Nov 01 00:24:53.493985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.493682 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerDied","Data":"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010"} Nov 01 00:24:53.918044 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:53.918011 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:54.127117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.127066 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:24:54.198024 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.197999 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.284750 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.284711 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.284953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.284768 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.284953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.284814 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.284953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.284866 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.284953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.284900 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.284953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.284925 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.285214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.284962 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.285214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.284990 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.285214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.285016 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.285214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.285062 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.285214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.285113 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.285214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.285142 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.285214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.285169 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web\") pod \"144d7167-16c0-4f04-a304-ebc22c47283e\" (UID: \"144d7167-16c0-4f04-a304-ebc22c47283e\") " Nov 01 00:24:54.285910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.285874 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:24:54.286778 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.286485 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:54.287964 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.287669 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:54.289753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.289653 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:54.289753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.289725 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:24:54.290199 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.290171 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:24:54.290316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.290169 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:54.290316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.290226 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "kube-api-access-r2q7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:24:54.290437 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.290335 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:54.290536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.290489 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:54.290632 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.290557 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:54.296631 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.296588 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:54.302709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.302679 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config" pod "144d7167-16c0-4f04-a304-ebc22c47283e" (UID: "144d7167-16c0-4f04-a304-ebc22c47283e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:54.386385 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386348 6036 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-main-db\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386385 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386378 6036 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386385 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386388 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-metrics-client-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386399 6036 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386408 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2q7j\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-kube-api-access-r2q7j\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386418 6036 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386427 6036 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/144d7167-16c0-4f04-a304-ebc22c47283e-config-out\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386435 6036 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/144d7167-16c0-4f04-a304-ebc22c47283e-tls-assets\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386446 6036 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144d7167-16c0-4f04-a304-ebc22c47283e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386455 6036 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-cluster-tls-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386464 6036 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-web-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386474 6036 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-main-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.386622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.386483 6036 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/144d7167-16c0-4f04-a304-ebc22c47283e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:54.502663 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.502577 6036 generic.go:356] "Generic (PLEG): container finished" podID="144d7167-16c0-4f04-a304-ebc22c47283e" containerID="8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d" exitCode=0 Nov 01 00:24:54.502663 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.502614 6036 generic.go:356] "Generic (PLEG): container finished" podID="144d7167-16c0-4f04-a304-ebc22c47283e" containerID="7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b" exitCode=0 Nov 01 00:24:54.503083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.502663 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerDied","Data":"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d"} Nov 01 00:24:54.503083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.502770 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerDied","Data":"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b"} Nov 01 00:24:54.503083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.502814 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"144d7167-16c0-4f04-a304-ebc22c47283e","Type":"ContainerDied","Data":"401f1de8b558af3b8ac80748f24fc3ea813e9e99a1fbf78be70e6a08dac1624f"} Nov 01 00:24:54.503083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.502865 6036 scope.go:122] "RemoveContainer" containerID="9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4" Nov 01 00:24:54.503083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.502690 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.512677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.512649 6036 scope.go:122] "RemoveContainer" containerID="8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d" Nov 01 00:24:54.520548 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.520527 6036 scope.go:122] "RemoveContainer" containerID="19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72" Nov 01 00:24:54.528340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.528313 6036 scope.go:122] "RemoveContainer" containerID="7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b" Nov 01 00:24:54.537113 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.537086 6036 scope.go:122] "RemoveContainer" containerID="087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1" Nov 01 00:24:54.545441 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.545423 6036 scope.go:122] "RemoveContainer" containerID="031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010" Nov 01 00:24:54.553308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.553262 6036 scope.go:122] "RemoveContainer" containerID="fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169" Nov 01 00:24:54.561123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.561107 6036 scope.go:122] "RemoveContainer" containerID="9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4" Nov 01 00:24:54.561446 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:54.561426 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4\": container with ID starting with 9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4 not found: ID does not exist" containerID="9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4" Nov 01 00:24:54.561501 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.561454 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4"} err="failed to get container status \"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4\": rpc error: code = NotFound desc = could not find container \"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4\": container with ID starting with 9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4 not found: ID does not exist" Nov 01 00:24:54.561501 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.561475 6036 scope.go:122] "RemoveContainer" containerID="8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d" Nov 01 00:24:54.561720 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:54.561704 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d\": container with ID starting with 8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d not found: ID does not exist" containerID="8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d" Nov 01 00:24:54.561765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.561728 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d"} err="failed to get container status \"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d\": rpc error: code = NotFound desc = could not find container \"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d\": container with ID starting with 8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d not found: ID does not exist" Nov 01 00:24:54.561765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.561745 6036 scope.go:122] "RemoveContainer" containerID="19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72" Nov 01 00:24:54.561946 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:54.561931 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72\": container with ID starting with 19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72 not found: ID does not exist" containerID="19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72" Nov 01 00:24:54.561994 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.561951 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72"} err="failed to get container status \"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72\": rpc error: code = NotFound desc = could not find container \"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72\": container with ID starting with 19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72 not found: ID does not exist" Nov 01 00:24:54.561994 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.561964 6036 scope.go:122] "RemoveContainer" containerID="7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b" Nov 01 00:24:54.562161 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:54.562146 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b\": container with ID starting with 7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b not found: ID does not exist" containerID="7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b" Nov 01 00:24:54.562197 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.562165 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b"} err="failed to get container status \"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b\": rpc error: code = NotFound desc = could not find container \"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b\": container with ID starting with 7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b not found: ID does not exist" Nov 01 00:24:54.562197 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.562176 6036 scope.go:122] "RemoveContainer" containerID="087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1" Nov 01 00:24:54.562419 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:54.562400 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1\": container with ID starting with 087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1 not found: ID does not exist" containerID="087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1" Nov 01 00:24:54.562471 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.562427 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1"} err="failed to get container status \"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1\": rpc error: code = NotFound desc = could not find container \"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1\": container with ID starting with 087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1 not found: ID does not exist" Nov 01 00:24:54.562471 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.562442 6036 scope.go:122] "RemoveContainer" containerID="031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010" Nov 01 00:24:54.562667 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:54.562653 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010\": container with ID starting with 031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010 not found: ID does not exist" containerID="031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010" Nov 01 00:24:54.562707 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.562671 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010"} err="failed to get container status \"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010\": rpc error: code = NotFound desc = could not find container \"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010\": container with ID starting with 031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010 not found: ID does not exist" Nov 01 00:24:54.562707 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.562684 6036 scope.go:122] "RemoveContainer" containerID="fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169" Nov 01 00:24:54.562887 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:54.562872 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169\": container with ID starting with fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169 not found: ID does not exist" containerID="fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169" Nov 01 00:24:54.562929 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.562891 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169"} err="failed to get container status \"fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169\": rpc error: code = NotFound desc = could not find container \"fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169\": container with ID starting with fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169 not found: ID does not exist" Nov 01 00:24:54.562929 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.562902 6036 scope.go:122] "RemoveContainer" containerID="9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4" Nov 01 00:24:54.563099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563076 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4"} err="failed to get container status \"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4\": rpc error: code = NotFound desc = could not find container \"9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4\": container with ID starting with 9b5061a0f814485710f8a54ad4bfa15b2f32fb0fa6cb6c139d131bf1082dd8f4 not found: ID does not exist" Nov 01 00:24:54.563139 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563099 6036 scope.go:122] "RemoveContainer" containerID="8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d" Nov 01 00:24:54.563299 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563267 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d"} err="failed to get container status \"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d\": rpc error: code = NotFound desc = could not find container \"8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d\": container with ID starting with 8b4cb7f10993ce0c04752714bdbdae8bf249c34fb8c6cb8fd7db138e6b29a55d not found: ID does not exist" Nov 01 00:24:54.563340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563300 6036 scope.go:122] "RemoveContainer" containerID="19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72" Nov 01 00:24:54.563504 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563488 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72"} err="failed to get container status \"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72\": rpc error: code = NotFound desc = could not find container \"19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72\": container with ID starting with 19b41361d6751f9fc1f0c8ce4f1877ca9979a4be0508bf8aa3f01bdd89d9cd72 not found: ID does not exist" Nov 01 00:24:54.563545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563503 6036 scope.go:122] "RemoveContainer" containerID="7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b" Nov 01 00:24:54.563709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563683 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b"} err="failed to get container status \"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b\": rpc error: code = NotFound desc = could not find container \"7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b\": container with ID starting with 7f517a1f253939219b84083b75c6b5e855140c43bd9cc3c6fac9d7033ed1a17b not found: ID does not exist" Nov 01 00:24:54.563709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563710 6036 scope.go:122] "RemoveContainer" containerID="087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1" Nov 01 00:24:54.563913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563896 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1"} err="failed to get container status \"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1\": rpc error: code = NotFound desc = could not find container \"087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1\": container with ID starting with 087a6ec64645d9fae59e572ef0f49cf00727591ddac0e60b2016302a95cdd7a1 not found: ID does not exist" Nov 01 00:24:54.563986 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.563914 6036 scope.go:122] "RemoveContainer" containerID="031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010" Nov 01 00:24:54.564137 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.564121 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010"} err="failed to get container status \"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010\": rpc error: code = NotFound desc = could not find container \"031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010\": container with ID starting with 031382a98efd9b109705e136cbaa0bc3d4e6d0f63ea323affa9461f4dd8c2010 not found: ID does not exist" Nov 01 00:24:54.564176 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.564137 6036 scope.go:122] "RemoveContainer" containerID="fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169" Nov 01 00:24:54.564347 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.564327 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169"} err="failed to get container status \"fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169\": rpc error: code = NotFound desc = could not find container \"fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169\": container with ID starting with fef650bcac96b7fa7910edf244c41c5ab40d23b9958bc49584a9b78980900169 not found: ID does not exist" Nov 01 00:24:54.675351 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.675318 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:24:54.678808 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.678788 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:24:54.725719 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.725692 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:24:54.726000 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.725989 6036 cpu_manager.go:414] "containerMap: removing container" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy-web" Nov 01 00:24:54.726050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726001 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy-web" Nov 01 00:24:54.726050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726016 6036 cpu_manager.go:414] "containerMap: removing container" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="config-reloader" Nov 01 00:24:54.726050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726021 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="config-reloader" Nov 01 00:24:54.726050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726028 6036 cpu_manager.go:414] "containerMap: removing container" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy" Nov 01 00:24:54.726050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726034 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy" Nov 01 00:24:54.726050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726040 6036 cpu_manager.go:414] "containerMap: removing container" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy-metric" Nov 01 00:24:54.726050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726045 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy-metric" Nov 01 00:24:54.726050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726051 6036 cpu_manager.go:414] "containerMap: removing container" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="alertmanager" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726056 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="alertmanager" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726062 6036 cpu_manager.go:414] "containerMap: removing container" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="init-config-reloader" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726067 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="init-config-reloader" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726076 6036 cpu_manager.go:414] "containerMap: removing container" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="prom-label-proxy" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726081 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="prom-label-proxy" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726092 6036 cpu_manager.go:414] "containerMap: removing container" podUID="7918db7e-5d6b-47bd-bd67-32811f8e40f7" containerName="console" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726097 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7918db7e-5d6b-47bd-bd67-32811f8e40f7" containerName="console" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726142 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy-web" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726149 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="config-reloader" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726154 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726159 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="kube-rbac-proxy-metric" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726165 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="alertmanager" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726170 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="init-config-reloader" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726175 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" containerName="prom-label-proxy" Nov 01 00:24:54.726273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.726182 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7918db7e-5d6b-47bd-bd67-32811f8e40f7" containerName="console" Nov 01 00:24:54.729407 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.729388 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.789873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.789843 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c292a777-bdf1-483f-945d-73c1ed2f6794-config-out\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.789889 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.789906 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c292a777-bdf1-483f-945d-73c1ed2f6794-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.789978 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790026 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c292a777-bdf1-483f-945d-73c1ed2f6794-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790168 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790069 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c292a777-bdf1-483f-945d-73c1ed2f6794-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790168 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790124 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790168 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790164 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790254 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790182 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c292a777-bdf1-483f-945d-73c1ed2f6794-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790254 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790235 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hdx\" (UniqueName: \"kubernetes.io/projected/c292a777-bdf1-483f-945d-73c1ed2f6794-kube-api-access-b4hdx\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790366 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790256 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-web-config\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790366 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790278 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.790366 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.790324 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-config-volume\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.792232 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792217 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 00:24:54.792486 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792455 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 00:24:54.792600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792537 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 00:24:54.792600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792543 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 00:24:54.792600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792540 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 00:24:54.792600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792587 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 00:24:54.792769 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792595 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 00:24:54.792822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792810 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8rsns\"" Nov 01 00:24:54.792853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.792814 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 00:24:54.842991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.842956 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 00:24:54.843728 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.843709 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:24:54.890883 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.890852 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.890883 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.890881 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c292a777-bdf1-483f-945d-73c1ed2f6794-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.890902 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.890922 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c292a777-bdf1-483f-945d-73c1ed2f6794-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.890938 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c292a777-bdf1-483f-945d-73c1ed2f6794-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.890956 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891121 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891311 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c292a777-bdf1-483f-945d-73c1ed2f6794-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891351 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hdx\" (UniqueName: \"kubernetes.io/projected/c292a777-bdf1-483f-945d-73c1ed2f6794-kube-api-access-b4hdx\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891385 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-web-config\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891427 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891461 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c292a777-bdf1-483f-945d-73c1ed2f6794-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891472 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-config-volume\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891648 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891501 6036 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 00:24:54.891791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891528 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c292a777-bdf1-483f-945d-73c1ed2f6794-config-out\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.891791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.891786 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c292a777-bdf1-483f-945d-73c1ed2f6794-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.892202 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.892156 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c292a777-bdf1-483f-945d-73c1ed2f6794-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.895167 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.895098 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.895167 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.895124 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c292a777-bdf1-483f-945d-73c1ed2f6794-config-out\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.895337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.895127 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-web-config\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.895337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.895196 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.895337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.895218 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c292a777-bdf1-483f-945d-73c1ed2f6794-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.895337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.895229 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.895461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.895344 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.895461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.895417 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.896503 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.896488 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c292a777-bdf1-483f-945d-73c1ed2f6794-config-volume\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:54.954911 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:54.954882 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hdx\" (UniqueName: \"kubernetes.io/projected/c292a777-bdf1-483f-945d-73c1ed2f6794-kube-api-access-b4hdx\") pod \"alertmanager-main-0\" (UID: \"c292a777-bdf1-483f-945d-73c1ed2f6794\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:55.045083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:55.044992 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 00:24:55.250915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:55.250880 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144d7167-16c0-4f04-a304-ebc22c47283e" path="/var/lib/kubelet/pods/144d7167-16c0-4f04-a304-ebc22c47283e/volumes" Nov 01 00:24:55.475004 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:55.474934 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 00:24:55.477729 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:24:55.477693 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc292a777_bdf1_483f_945d_73c1ed2f6794.slice/crio-166722a3776e2152357a0a74a218bc9bfabc3a77de99a6f9b03592b81b77dfd3 WatchSource:0}: Error finding container 166722a3776e2152357a0a74a218bc9bfabc3a77de99a6f9b03592b81b77dfd3: Status 404 returned error can't find the container with id 166722a3776e2152357a0a74a218bc9bfabc3a77de99a6f9b03592b81b77dfd3 Nov 01 00:24:55.507882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:55.507848 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c292a777-bdf1-483f-945d-73c1ed2f6794","Type":"ContainerStarted","Data":"166722a3776e2152357a0a74a218bc9bfabc3a77de99a6f9b03592b81b77dfd3"} Nov 01 00:24:56.514620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:56.514590 6036 generic.go:356] "Generic (PLEG): container finished" podID="c292a777-bdf1-483f-945d-73c1ed2f6794" containerID="5bac6fdc9cf4a24275e726086beaf67c1c2f75d5a26b6cfa98b9e9b1ce264b58" exitCode=0 Nov 01 00:24:56.515013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:56.514658 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c292a777-bdf1-483f-945d-73c1ed2f6794","Type":"ContainerDied","Data":"5bac6fdc9cf4a24275e726086beaf67c1c2f75d5a26b6cfa98b9e9b1ce264b58"} Nov 01 00:24:57.526051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:57.526010 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c292a777-bdf1-483f-945d-73c1ed2f6794","Type":"ContainerStarted","Data":"8eec391c8f062c455db088053ccff3430b1a5c2b3e7fdaa69844b0b849e45c9d"} Nov 01 00:24:57.526476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:57.526123 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c292a777-bdf1-483f-945d-73c1ed2f6794","Type":"ContainerStarted","Data":"b71a567f052e601df6f4f71dc7b51c46e8e95d6e4f9ab3256a957445b1351143"} Nov 01 00:24:57.526476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:57.526164 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c292a777-bdf1-483f-945d-73c1ed2f6794","Type":"ContainerStarted","Data":"bd3e4f7d213b15278f001afb16c4b32769c814461300638c8ef493d1b03d553c"} Nov 01 00:24:57.526476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:57.526206 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c292a777-bdf1-483f-945d-73c1ed2f6794","Type":"ContainerStarted","Data":"86c4fc3aefc1722d85a0c2600148d6376ee859265fc33bc735c700791a2b587d"} Nov 01 00:24:57.526476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:57.526239 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c292a777-bdf1-483f-945d-73c1ed2f6794","Type":"ContainerStarted","Data":"61a0a33f8eb4068af250cb1b9ad3956c07a44c02d3e27e117e65393868fb715d"} Nov 01 00:24:57.526476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:57.526274 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c292a777-bdf1-483f-945d-73c1ed2f6794","Type":"ContainerStarted","Data":"2194919320aabbfe2659a15037bde9c6ae57f8565b06df0f118304bbd517608c"} Nov 01 00:24:57.657307 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:57.657229 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.645219239 podStartE2EDuration="3.657215896s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.996657ms" isStatefulPod=false podCreationTimestamp="2026-11-01 00:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:57.656861079 +0000 UTC m=+42.532082107" watchObservedRunningTime="2026-11-01 00:24:57.657215896 +0000 UTC m=+42.532436908" Nov 01 00:24:58.691108 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:58.691068 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:24:58.691609 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:58.691478 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="prometheus" containerID="cri-o://b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" gracePeriod=600 Nov 01 00:24:58.691609 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:58.691532 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy-thanos" containerID="cri-o://67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" gracePeriod=600 Nov 01 00:24:58.691609 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:58.691535 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="thanos-sidecar" containerID="cri-o://ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" gracePeriod=600 Nov 01 00:24:58.691839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:58.691515 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy" containerID="cri-o://6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" gracePeriod=600 Nov 01 00:24:58.691839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:58.691565 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="config-reloader" containerID="cri-o://79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" gracePeriod=600 Nov 01 00:24:58.691839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:58.691535 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy-web" containerID="cri-o://644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" gracePeriod=600 Nov 01 00:24:58.955909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:58.955881 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.030957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.030924 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.030957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.030970 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031228 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031008 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031228 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031035 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031228 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031153 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031228 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031218 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031252 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031377 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031429 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031476 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031519 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031547 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031591 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.031664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031627 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.032035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031666 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.032035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031697 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.032035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031727 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.032035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031758 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.032035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031789 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.032035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.031860 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db\") pod \"186510bd-08db-41ac-9e29-4907eb505d01\" (UID: \"186510bd-08db-41ac-9e29-4907eb505d01\") " Nov 01 00:24:59.033075 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.033015 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:24:59.033985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.033839 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:59.033985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.033862 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:59.034392 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.034181 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:59.034392 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.034331 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:59.034392 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.034338 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:59.035054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.034636 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:59.035399 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.035264 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:59.035399 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.035339 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.035545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.035499 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.035686 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.035664 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.035768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.035748 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:24:59.035923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.035885 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.036047 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.036026 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.036524 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.036492 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:24:59.036631 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.036596 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.036718 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.036698 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.036939 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.036917 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "kube-api-access-5g85t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:24:59.036939 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.036922 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.047441 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.047414 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config" pod "186510bd-08db-41ac-9e29-4907eb505d01" (UID: "186510bd-08db-41ac-9e29-4907eb505d01"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:59.132971 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.132930 6036 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.132971 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.132961 6036 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.132971 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.132975 6036 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.132990 6036 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-metrics-client-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133003 6036 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-db\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133016 6036 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133027 6036 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-grpc-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133039 6036 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133053 6036 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133065 6036 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/186510bd-08db-41ac-9e29-4907eb505d01-config-out\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133078 6036 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-tls-assets\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133092 6036 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133105 6036 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133119 6036 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-web-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133132 6036 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133144 6036 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-metrics-client-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133162 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g85t\" (UniqueName: \"kubernetes.io/projected/186510bd-08db-41ac-9e29-4907eb505d01-kube-api-access-5g85t\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133177 6036 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133192 6036 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/186510bd-08db-41ac-9e29-4907eb505d01-secret-kube-rbac-proxy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.133260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.133208 6036 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/186510bd-08db-41ac-9e29-4907eb505d01-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:59.537187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537150 6036 generic.go:356] "Generic (PLEG): container finished" podID="186510bd-08db-41ac-9e29-4907eb505d01" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" exitCode=0 Nov 01 00:24:59.537187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537174 6036 generic.go:356] "Generic (PLEG): container finished" podID="186510bd-08db-41ac-9e29-4907eb505d01" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" exitCode=0 Nov 01 00:24:59.537187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537180 6036 generic.go:356] "Generic (PLEG): container finished" podID="186510bd-08db-41ac-9e29-4907eb505d01" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" exitCode=0 Nov 01 00:24:59.537187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537185 6036 generic.go:356] "Generic (PLEG): container finished" podID="186510bd-08db-41ac-9e29-4907eb505d01" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" exitCode=0 Nov 01 00:24:59.537187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537192 6036 generic.go:356] "Generic (PLEG): container finished" podID="186510bd-08db-41ac-9e29-4907eb505d01" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" exitCode=0 Nov 01 00:24:59.537187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537197 6036 generic.go:356] "Generic (PLEG): container finished" podID="186510bd-08db-41ac-9e29-4907eb505d01" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" exitCode=0 Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537216 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerDied","Data":"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065"} Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537237 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537304 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerDied","Data":"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643"} Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537320 6036 scope.go:122] "RemoveContainer" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537343 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerDied","Data":"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1"} Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537368 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerDied","Data":"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b"} Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537398 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerDied","Data":"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c"} Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537427 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerDied","Data":"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415"} Nov 01 00:24:59.537560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.537465 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"186510bd-08db-41ac-9e29-4907eb505d01","Type":"ContainerDied","Data":"b8a27a8342fc5c9c161f80120b904e045b4ad2f50dd27b7f0e8b05daaac643a4"} Nov 01 00:24:59.545102 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.545071 6036 scope.go:122] "RemoveContainer" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" Nov 01 00:24:59.552598 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.552574 6036 scope.go:122] "RemoveContainer" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" Nov 01 00:24:59.559930 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.559911 6036 scope.go:122] "RemoveContainer" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" Nov 01 00:24:59.569547 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.569442 6036 scope.go:122] "RemoveContainer" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" Nov 01 00:24:59.579845 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.579825 6036 scope.go:122] "RemoveContainer" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" Nov 01 00:24:59.588498 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.588476 6036 scope.go:122] "RemoveContainer" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" Nov 01 00:24:59.599066 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.599037 6036 scope.go:122] "RemoveContainer" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" Nov 01 00:24:59.599391 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:59.599360 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": container with ID starting with 67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065 not found: ID does not exist" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" Nov 01 00:24:59.599461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.599405 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065"} err="failed to get container status \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": rpc error: code = NotFound desc = could not find container \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": container with ID starting with 67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065 not found: ID does not exist" Nov 01 00:24:59.599461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.599425 6036 scope.go:122] "RemoveContainer" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" Nov 01 00:24:59.599688 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:59.599672 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": container with ID starting with 6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643 not found: ID does not exist" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" Nov 01 00:24:59.599725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.599698 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643"} err="failed to get container status \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": rpc error: code = NotFound desc = could not find container \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": container with ID starting with 6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643 not found: ID does not exist" Nov 01 00:24:59.599725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.599720 6036 scope.go:122] "RemoveContainer" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" Nov 01 00:24:59.599972 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:59.599957 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": container with ID starting with 644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1 not found: ID does not exist" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" Nov 01 00:24:59.600019 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.599976 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1"} err="failed to get container status \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": rpc error: code = NotFound desc = could not find container \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": container with ID starting with 644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1 not found: ID does not exist" Nov 01 00:24:59.600019 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.599988 6036 scope.go:122] "RemoveContainer" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" Nov 01 00:24:59.600235 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:59.600205 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": container with ID starting with ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b not found: ID does not exist" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" Nov 01 00:24:59.600304 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.600239 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b"} err="failed to get container status \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": rpc error: code = NotFound desc = could not find container \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": container with ID starting with ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b not found: ID does not exist" Nov 01 00:24:59.600304 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.600257 6036 scope.go:122] "RemoveContainer" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" Nov 01 00:24:59.600515 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:59.600500 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": container with ID starting with 79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c not found: ID does not exist" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" Nov 01 00:24:59.600576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.600518 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c"} err="failed to get container status \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": rpc error: code = NotFound desc = could not find container \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": container with ID starting with 79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c not found: ID does not exist" Nov 01 00:24:59.600576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.600531 6036 scope.go:122] "RemoveContainer" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" Nov 01 00:24:59.600797 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:59.600779 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": container with ID starting with b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415 not found: ID does not exist" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" Nov 01 00:24:59.600846 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.600801 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415"} err="failed to get container status \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": rpc error: code = NotFound desc = could not find container \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": container with ID starting with b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415 not found: ID does not exist" Nov 01 00:24:59.600846 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.600822 6036 scope.go:122] "RemoveContainer" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" Nov 01 00:24:59.601076 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:24:59.601060 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": container with ID starting with 39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4 not found: ID does not exist" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" Nov 01 00:24:59.601117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.601089 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4"} err="failed to get container status \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": rpc error: code = NotFound desc = could not find container \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": container with ID starting with 39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4 not found: ID does not exist" Nov 01 00:24:59.601117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.601102 6036 scope.go:122] "RemoveContainer" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" Nov 01 00:24:59.601364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.601346 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065"} err="failed to get container status \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": rpc error: code = NotFound desc = could not find container \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": container with ID starting with 67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065 not found: ID does not exist" Nov 01 00:24:59.601364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.601363 6036 scope.go:122] "RemoveContainer" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" Nov 01 00:24:59.603538 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.603509 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643"} err="failed to get container status \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": rpc error: code = NotFound desc = could not find container \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": container with ID starting with 6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643 not found: ID does not exist" Nov 01 00:24:59.603538 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.603538 6036 scope.go:122] "RemoveContainer" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" Nov 01 00:24:59.604253 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.604232 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1"} err="failed to get container status \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": rpc error: code = NotFound desc = could not find container \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": container with ID starting with 644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1 not found: ID does not exist" Nov 01 00:24:59.604362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.604254 6036 scope.go:122] "RemoveContainer" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" Nov 01 00:24:59.604606 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.604582 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b"} err="failed to get container status \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": rpc error: code = NotFound desc = could not find container \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": container with ID starting with ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b not found: ID does not exist" Nov 01 00:24:59.604668 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.604609 6036 scope.go:122] "RemoveContainer" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" Nov 01 00:24:59.605045 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.604896 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c"} err="failed to get container status \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": rpc error: code = NotFound desc = could not find container \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": container with ID starting with 79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c not found: ID does not exist" Nov 01 00:24:59.605045 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.604921 6036 scope.go:122] "RemoveContainer" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" Nov 01 00:24:59.605257 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.605239 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415"} err="failed to get container status \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": rpc error: code = NotFound desc = could not find container \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": container with ID starting with b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415 not found: ID does not exist" Nov 01 00:24:59.605353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.605258 6036 scope.go:122] "RemoveContainer" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" Nov 01 00:24:59.605550 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.605532 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4"} err="failed to get container status \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": rpc error: code = NotFound desc = could not find container \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": container with ID starting with 39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4 not found: ID does not exist" Nov 01 00:24:59.605637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.605552 6036 scope.go:122] "RemoveContainer" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" Nov 01 00:24:59.605842 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.605816 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065"} err="failed to get container status \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": rpc error: code = NotFound desc = could not find container \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": container with ID starting with 67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065 not found: ID does not exist" Nov 01 00:24:59.605884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.605843 6036 scope.go:122] "RemoveContainer" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" Nov 01 00:24:59.606118 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.606090 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643"} err="failed to get container status \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": rpc error: code = NotFound desc = could not find container \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": container with ID starting with 6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643 not found: ID does not exist" Nov 01 00:24:59.606169 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.606119 6036 scope.go:122] "RemoveContainer" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" Nov 01 00:24:59.606400 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.606381 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1"} err="failed to get container status \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": rpc error: code = NotFound desc = could not find container \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": container with ID starting with 644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1 not found: ID does not exist" Nov 01 00:24:59.606481 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.606401 6036 scope.go:122] "RemoveContainer" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" Nov 01 00:24:59.606647 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.606629 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b"} err="failed to get container status \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": rpc error: code = NotFound desc = could not find container \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": container with ID starting with ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b not found: ID does not exist" Nov 01 00:24:59.606699 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.606647 6036 scope.go:122] "RemoveContainer" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" Nov 01 00:24:59.606824 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.606804 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c"} err="failed to get container status \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": rpc error: code = NotFound desc = could not find container \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": container with ID starting with 79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c not found: ID does not exist" Nov 01 00:24:59.606872 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.606824 6036 scope.go:122] "RemoveContainer" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" Nov 01 00:24:59.607047 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.607023 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415"} err="failed to get container status \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": rpc error: code = NotFound desc = could not find container \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": container with ID starting with b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415 not found: ID does not exist" Nov 01 00:24:59.607098 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.607052 6036 scope.go:122] "RemoveContainer" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" Nov 01 00:24:59.607356 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.607333 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4"} err="failed to get container status \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": rpc error: code = NotFound desc = could not find container \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": container with ID starting with 39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4 not found: ID does not exist" Nov 01 00:24:59.607356 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.607354 6036 scope.go:122] "RemoveContainer" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" Nov 01 00:24:59.607616 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.607592 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065"} err="failed to get container status \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": rpc error: code = NotFound desc = could not find container \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": container with ID starting with 67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065 not found: ID does not exist" Nov 01 00:24:59.607616 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.607615 6036 scope.go:122] "RemoveContainer" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" Nov 01 00:24:59.607851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.607825 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643"} err="failed to get container status \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": rpc error: code = NotFound desc = could not find container \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": container with ID starting with 6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643 not found: ID does not exist" Nov 01 00:24:59.607896 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.607853 6036 scope.go:122] "RemoveContainer" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" Nov 01 00:24:59.608129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.608112 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1"} err="failed to get container status \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": rpc error: code = NotFound desc = could not find container \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": container with ID starting with 644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1 not found: ID does not exist" Nov 01 00:24:59.608185 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.608130 6036 scope.go:122] "RemoveContainer" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" Nov 01 00:24:59.608379 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.608362 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b"} err="failed to get container status \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": rpc error: code = NotFound desc = could not find container \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": container with ID starting with ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b not found: ID does not exist" Nov 01 00:24:59.608445 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.608380 6036 scope.go:122] "RemoveContainer" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" Nov 01 00:24:59.608618 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.608595 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c"} err="failed to get container status \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": rpc error: code = NotFound desc = could not find container \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": container with ID starting with 79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c not found: ID does not exist" Nov 01 00:24:59.608618 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.608616 6036 scope.go:122] "RemoveContainer" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" Nov 01 00:24:59.608938 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.608914 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415"} err="failed to get container status \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": rpc error: code = NotFound desc = could not find container \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": container with ID starting with b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415 not found: ID does not exist" Nov 01 00:24:59.608938 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.608937 6036 scope.go:122] "RemoveContainer" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" Nov 01 00:24:59.609270 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.609240 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4"} err="failed to get container status \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": rpc error: code = NotFound desc = could not find container \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": container with ID starting with 39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4 not found: ID does not exist" Nov 01 00:24:59.609270 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.609261 6036 scope.go:122] "RemoveContainer" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" Nov 01 00:24:59.609645 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.609624 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065"} err="failed to get container status \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": rpc error: code = NotFound desc = could not find container \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": container with ID starting with 67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065 not found: ID does not exist" Nov 01 00:24:59.609735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.609646 6036 scope.go:122] "RemoveContainer" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" Nov 01 00:24:59.609889 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.609870 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643"} err="failed to get container status \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": rpc error: code = NotFound desc = could not find container \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": container with ID starting with 6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643 not found: ID does not exist" Nov 01 00:24:59.609978 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.609890 6036 scope.go:122] "RemoveContainer" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" Nov 01 00:24:59.610141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.610123 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1"} err="failed to get container status \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": rpc error: code = NotFound desc = could not find container \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": container with ID starting with 644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1 not found: ID does not exist" Nov 01 00:24:59.610226 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.610142 6036 scope.go:122] "RemoveContainer" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" Nov 01 00:24:59.610443 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.610424 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b"} err="failed to get container status \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": rpc error: code = NotFound desc = could not find container \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": container with ID starting with ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b not found: ID does not exist" Nov 01 00:24:59.610518 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.610444 6036 scope.go:122] "RemoveContainer" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" Nov 01 00:24:59.610684 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.610666 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c"} err="failed to get container status \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": rpc error: code = NotFound desc = could not find container \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": container with ID starting with 79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c not found: ID does not exist" Nov 01 00:24:59.610768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.610685 6036 scope.go:122] "RemoveContainer" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" Nov 01 00:24:59.610930 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.610910 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415"} err="failed to get container status \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": rpc error: code = NotFound desc = could not find container \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": container with ID starting with b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415 not found: ID does not exist" Nov 01 00:24:59.611009 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.610931 6036 scope.go:122] "RemoveContainer" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" Nov 01 00:24:59.611219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.611201 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4"} err="failed to get container status \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": rpc error: code = NotFound desc = could not find container \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": container with ID starting with 39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4 not found: ID does not exist" Nov 01 00:24:59.611321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.611220 6036 scope.go:122] "RemoveContainer" containerID="67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065" Nov 01 00:24:59.611530 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.611504 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065"} err="failed to get container status \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": rpc error: code = NotFound desc = could not find container \"67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065\": container with ID starting with 67ebb2364d03d953ea94a13c71d735516479d8881a997fd571cdf34f6bda8065 not found: ID does not exist" Nov 01 00:24:59.611530 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.611530 6036 scope.go:122] "RemoveContainer" containerID="6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643" Nov 01 00:24:59.611852 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.611828 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643"} err="failed to get container status \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": rpc error: code = NotFound desc = could not find container \"6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643\": container with ID starting with 6000d23eedf3ff2205330c1911eebc34604360f0983c65621c446fbc74b79643 not found: ID does not exist" Nov 01 00:24:59.611852 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.611851 6036 scope.go:122] "RemoveContainer" containerID="644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1" Nov 01 00:24:59.612120 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.612099 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1"} err="failed to get container status \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": rpc error: code = NotFound desc = could not find container \"644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1\": container with ID starting with 644c95ea6f7718c612cb94809ef9fd0189a4f705a0c12f75cb3371921e89cfb1 not found: ID does not exist" Nov 01 00:24:59.612194 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.612121 6036 scope.go:122] "RemoveContainer" containerID="ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b" Nov 01 00:24:59.612421 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.612402 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b"} err="failed to get container status \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": rpc error: code = NotFound desc = could not find container \"ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b\": container with ID starting with ffdf3d5fd5fa3d9b289b59316b7e2b1602b54912b970e5a19743e6215ba4ea5b not found: ID does not exist" Nov 01 00:24:59.612421 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.612420 6036 scope.go:122] "RemoveContainer" containerID="79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c" Nov 01 00:24:59.612643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.612620 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c"} err="failed to get container status \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": rpc error: code = NotFound desc = could not find container \"79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c\": container with ID starting with 79f3874168762c0e431871075a29ea268664aeaf083df521fe2c79c070d9334c not found: ID does not exist" Nov 01 00:24:59.612643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.612637 6036 scope.go:122] "RemoveContainer" containerID="b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415" Nov 01 00:24:59.612936 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.612909 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415"} err="failed to get container status \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": rpc error: code = NotFound desc = could not find container \"b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415\": container with ID starting with b570c07ec3c9412b90b9c680811f36d912129c2c94f62974384aecb18a4e9415 not found: ID does not exist" Nov 01 00:24:59.613010 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.612936 6036 scope.go:122] "RemoveContainer" containerID="39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4" Nov 01 00:24:59.613180 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.613158 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4"} err="failed to get container status \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": rpc error: code = NotFound desc = could not find container \"39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4\": container with ID starting with 39b53a4a8a6b2b9f31023686201b7226d3e088d3585eee5e27b023de9e196ba4 not found: ID does not exist" Nov 01 00:24:59.709085 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.709044 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:24:59.759583 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.759553 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:24:59.759906 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.759885 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:24:59.760179 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760167 6036 cpu_manager.go:414] "containerMap: removing container" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy-thanos" Nov 01 00:24:59.760215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760181 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy-thanos" Nov 01 00:24:59.760215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760194 6036 cpu_manager.go:414] "containerMap: removing container" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="prometheus" Nov 01 00:24:59.760215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760200 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="prometheus" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760217 6036 cpu_manager.go:414] "containerMap: removing container" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="thanos-sidecar" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760222 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="thanos-sidecar" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760231 6036 cpu_manager.go:414] "containerMap: removing container" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760237 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760243 6036 cpu_manager.go:414] "containerMap: removing container" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="config-reloader" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760247 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="config-reloader" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760258 6036 cpu_manager.go:414] "containerMap: removing container" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="init-config-reloader" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760263 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="init-config-reloader" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760270 6036 cpu_manager.go:414] "containerMap: removing container" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy-web" Nov 01 00:24:59.760318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760274 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy-web" Nov 01 00:24:59.760588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760335 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy" Nov 01 00:24:59.760588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760343 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="config-reloader" Nov 01 00:24:59.760588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760349 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="init-config-reloader" Nov 01 00:24:59.760588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760355 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy-web" Nov 01 00:24:59.760588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760362 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="kube-rbac-proxy-thanos" Nov 01 00:24:59.760588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760367 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="prometheus" Nov 01 00:24:59.760588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.760373 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="186510bd-08db-41ac-9e29-4907eb505d01" containerName="thanos-sidecar" Nov 01 00:24:59.766065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.766036 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.819694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819535 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 00:24:59.819694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819557 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 00:24:59.819694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819543 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 00:24:59.819694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819657 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 00:24:59.819694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819698 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 00:24:59.819992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819721 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7rnajk1nfvaac\"" Nov 01 00:24:59.819992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819756 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 00:24:59.819992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819769 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 00:24:59.819992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819774 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j7jdw\"" Nov 01 00:24:59.819992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819816 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 00:24:59.819992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.819817 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 00:24:59.826378 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.826355 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 00:24:59.826503 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.826412 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 00:24:59.826563 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.826519 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 00:24:59.837673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837644 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/866170d7-d2d3-4cff-9266-d9ff163fc2d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.837770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837683 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.837770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837715 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/866170d7-d2d3-4cff-9266-d9ff163fc2d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.837853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837817 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.837853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837840 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-config\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.837925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837857 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.837925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837872 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.837925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837897 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837934 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.837989 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838088 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838030 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838122 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838174 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838150 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838171 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838188 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838212 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838339 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838271 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838339 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838313 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zpt\" (UniqueName: \"kubernetes.io/projected/866170d7-d2d3-4cff-9266-d9ff163fc2d9-kube-api-access-n6zpt\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838337 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.838419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.838401 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.877713 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.877681 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 00:24:59.877854 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.877807 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 00:24:59.881426 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.881403 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:24:59.939624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939591 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/866170d7-d2d3-4cff-9266-d9ff163fc2d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939634 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939659 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/866170d7-d2d3-4cff-9266-d9ff163fc2d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939688 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939708 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-config\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939730 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939744 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939759 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939782 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939809 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939832 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.939890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939875 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939901 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939931 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939954 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.939974 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.940027 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.940051 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zpt\" (UniqueName: \"kubernetes.io/projected/866170d7-d2d3-4cff-9266-d9ff163fc2d9-kube-api-access-n6zpt\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.940079 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.940110 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.940812 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.940776 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.941169 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.940972 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.942863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.941903 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.942863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.941931 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.942863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.942627 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.943744 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.943188 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.943744 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.943352 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.943744 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.943432 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.943744 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.943513 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.944006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.943798 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.944006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.943902 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-config\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.944514 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.944470 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/866170d7-d2d3-4cff-9266-d9ff163fc2d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.944644 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.944475 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/866170d7-d2d3-4cff-9266-d9ff163fc2d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.945527 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.945500 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.945719 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.945697 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/866170d7-d2d3-4cff-9266-d9ff163fc2d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.945790 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.945748 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.946121 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.946099 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.946342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.946320 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:24:59.946661 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:24:59.946640 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/866170d7-d2d3-4cff-9266-d9ff163fc2d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:25:00.007030 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:00.006995 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zpt\" (UniqueName: \"kubernetes.io/projected/866170d7-d2d3-4cff-9266-d9ff163fc2d9-kube-api-access-n6zpt\") pod \"prometheus-k8s-0\" (UID: \"866170d7-d2d3-4cff-9266-d9ff163fc2d9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:25:00.082874 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:00.082786 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:25:00.514321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:00.514225 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 00:25:00.517617 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:25:00.517587 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866170d7_d2d3_4cff_9266_d9ff163fc2d9.slice/crio-3d715e4c66d2a1a978d3eacdc5b88d893caecb180a4ce1304991113d719c0511 WatchSource:0}: Error finding container 3d715e4c66d2a1a978d3eacdc5b88d893caecb180a4ce1304991113d719c0511: Status 404 returned error can't find the container with id 3d715e4c66d2a1a978d3eacdc5b88d893caecb180a4ce1304991113d719c0511 Nov 01 00:25:00.542518 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:00.542491 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"866170d7-d2d3-4cff-9266-d9ff163fc2d9","Type":"ContainerStarted","Data":"3d715e4c66d2a1a978d3eacdc5b88d893caecb180a4ce1304991113d719c0511"} Nov 01 00:25:01.248788 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:01.248754 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186510bd-08db-41ac-9e29-4907eb505d01" path="/var/lib/kubelet/pods/186510bd-08db-41ac-9e29-4907eb505d01/volumes" Nov 01 00:25:01.551546 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:01.551514 6036 generic.go:356] "Generic (PLEG): container finished" podID="866170d7-d2d3-4cff-9266-d9ff163fc2d9" containerID="ebaf4be9216abb79b6c7c38496b260d6a6ede7ee4db1cacd1a45eb8542b5b93e" exitCode=0 Nov 01 00:25:01.551697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:01.551554 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"866170d7-d2d3-4cff-9266-d9ff163fc2d9","Type":"ContainerDied","Data":"ebaf4be9216abb79b6c7c38496b260d6a6ede7ee4db1cacd1a45eb8542b5b93e"} Nov 01 00:25:02.487522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.487486 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66c4c47bfd-n46cz"] Nov 01 00:25:02.489697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.489678 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.551988 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.551947 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c4c47bfd-n46cz"] Nov 01 00:25:02.559306 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.559251 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"866170d7-d2d3-4cff-9266-d9ff163fc2d9","Type":"ContainerStarted","Data":"b0299abf6c561970d1632185f26ea8a8ee76c806953bb58240f9944e4b1d9404"} Nov 01 00:25:02.559453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.559335 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"866170d7-d2d3-4cff-9266-d9ff163fc2d9","Type":"ContainerStarted","Data":"b6351ca8cc016182fde91ba3cc32d20a0b243c707a1b7c4c56fbef5654ef7ef8"} Nov 01 00:25:02.559453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.559362 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"866170d7-d2d3-4cff-9266-d9ff163fc2d9","Type":"ContainerStarted","Data":"446b3c5cb0db3a5fe5abc9940c1dea2985f6cfcec5d92034ded2bf8def482a21"} Nov 01 00:25:02.559453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.559385 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"866170d7-d2d3-4cff-9266-d9ff163fc2d9","Type":"ContainerStarted","Data":"1908bf3f558cebf02eddc128dc56205920a48912b245a9db7d36b6c72cfc152f"} Nov 01 00:25:02.559453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.559412 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"866170d7-d2d3-4cff-9266-d9ff163fc2d9","Type":"ContainerStarted","Data":"66eaef5994161ff0a5807dc81b25f8c5e2c8d0646d148ffbd6eeb7f1a81fe697"} Nov 01 00:25:02.559453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.559440 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"866170d7-d2d3-4cff-9266-d9ff163fc2d9","Type":"ContainerStarted","Data":"c10dd8480dbdad6a9f3c9ac282e6b953ef77d70faa2b39663c57276b69cf7c6c"} Nov 01 00:25:02.565991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.565963 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/365c1b95-a563-4310-82f2-894d776e88cd-tmp\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.566105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.566005 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-service-ca\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.566105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.566027 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-console-config\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.566183 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.566125 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-trusted-ca-bundle\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.566183 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.566161 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fkfj\" (UniqueName: \"kubernetes.io/projected/365c1b95-a563-4310-82f2-894d776e88cd-kube-api-access-8fkfj\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.566251 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.566205 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-serving-cert\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.566251 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.566232 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-oauth-config\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.566367 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.566276 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-oauth-serving-cert\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.667240 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.667200 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-service-ca\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.667240 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.667237 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-console-config\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.667517 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.667258 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-trusted-ca-bundle\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.667517 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.667310 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fkfj\" (UniqueName: \"kubernetes.io/projected/365c1b95-a563-4310-82f2-894d776e88cd-kube-api-access-8fkfj\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.667517 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.667374 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-serving-cert\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.668643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.667818 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-oauth-config\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.668643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.667858 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-oauth-serving-cert\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.668643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.668084 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/365c1b95-a563-4310-82f2-894d776e88cd-tmp\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.668643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.668177 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-console-config\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.668643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.668086 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-service-ca\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.668643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.668553 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/365c1b95-a563-4310-82f2-894d776e88cd-tmp\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.668643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.668571 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-oauth-serving-cert\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.669104 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.669081 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-trusted-ca-bundle\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.670584 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.670557 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-oauth-config\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.670677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.670585 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-serving-cert\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.729065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.729032 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fkfj\" (UniqueName: \"kubernetes.io/projected/365c1b95-a563-4310-82f2-894d776e88cd-kube-api-access-8fkfj\") pod \"console-66c4c47bfd-n46cz\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.745970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.745849 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.72820319 podStartE2EDuration="3.745829986s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.626796ms" isStatefulPod=false podCreationTimestamp="2026-11-01 00:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:25:02.745663607 +0000 UTC m=+47.620884623" watchObservedRunningTime="2026-11-01 00:25:02.745829986 +0000 UTC m=+47.621050996" Nov 01 00:25:02.800422 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.800391 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:02.831057 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:02.831026 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:25:03.042904 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:03.042849 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:25:03.184245 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:03.184212 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c4c47bfd-n46cz"] Nov 01 00:25:03.187149 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:25:03.187118 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365c1b95_a563_4310_82f2_894d776e88cd.slice/crio-c20ba0721f77e8b98d6a7758cfb222b0e3053c22955722da73b3f51fb44742a4 WatchSource:0}: Error finding container c20ba0721f77e8b98d6a7758cfb222b0e3053c22955722da73b3f51fb44742a4: Status 404 returned error can't find the container with id c20ba0721f77e8b98d6a7758cfb222b0e3053c22955722da73b3f51fb44742a4 Nov 01 00:25:03.309886 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:03.309759 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:25:03.565044 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:03.564948 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c4c47bfd-n46cz" event={"ID":"365c1b95-a563-4310-82f2-894d776e88cd","Type":"ContainerStarted","Data":"a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f"} Nov 01 00:25:03.565425 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:03.565043 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c4c47bfd-n46cz" event={"ID":"365c1b95-a563-4310-82f2-894d776e88cd","Type":"ContainerStarted","Data":"c20ba0721f77e8b98d6a7758cfb222b0e3053c22955722da73b3f51fb44742a4"} Nov 01 00:25:03.685166 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:03.685117 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66c4c47bfd-n46cz" podStartSLOduration=1.685103571 podStartE2EDuration="1.685103571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:25:03.684758294 +0000 UTC m=+48.559979310" watchObservedRunningTime="2026-11-01 00:25:03.685103571 +0000 UTC m=+48.560324577" Nov 01 00:25:05.083923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:05.083878 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:25:12.800585 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:12.800552 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:12.800968 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:12.800608 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:12.805517 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:12.805494 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:13.606121 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:13.606088 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:25:13.778186 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:13.778156 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585bf7664f-h64rc"] Nov 01 00:25:19.502649 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:25:19.502608 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:25:20.351906 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:20.351867 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:25:36.205684 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:36.205656 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:25:36.434695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:36.434663 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:25:38.798279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:38.798220 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-585bf7664f-h64rc" podUID="33e31e6c-2296-4e75-9eed-be8579a717b9" containerName="console" containerID="cri-o://b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077" gracePeriod=15 Nov 01 00:25:39.052516 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.052452 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:25:39.103707 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.103673 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config\") pod \"33e31e6c-2296-4e75-9eed-be8579a717b9\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " Nov 01 00:25:39.103873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.103719 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca\") pod \"33e31e6c-2296-4e75-9eed-be8579a717b9\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " Nov 01 00:25:39.103873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.103752 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp\") pod \"33e31e6c-2296-4e75-9eed-be8579a717b9\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " Nov 01 00:25:39.103873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.103805 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config\") pod \"33e31e6c-2296-4e75-9eed-be8579a717b9\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " Nov 01 00:25:39.103873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.103867 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh\" (UniqueName: \"kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh\") pod \"33e31e6c-2296-4e75-9eed-be8579a717b9\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " Nov 01 00:25:39.104082 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.103893 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert\") pod \"33e31e6c-2296-4e75-9eed-be8579a717b9\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " Nov 01 00:25:39.104082 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.103958 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle\") pod \"33e31e6c-2296-4e75-9eed-be8579a717b9\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " Nov 01 00:25:39.104082 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.103990 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert\") pod \"33e31e6c-2296-4e75-9eed-be8579a717b9\" (UID: \"33e31e6c-2296-4e75-9eed-be8579a717b9\") " Nov 01 00:25:39.104240 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.104128 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp" pod "33e31e6c-2296-4e75-9eed-be8579a717b9" (UID: "33e31e6c-2296-4e75-9eed-be8579a717b9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:25:39.104240 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.104181 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca" pod "33e31e6c-2296-4e75-9eed-be8579a717b9" (UID: "33e31e6c-2296-4e75-9eed-be8579a717b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:25:39.104384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.104316 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config" pod "33e31e6c-2296-4e75-9eed-be8579a717b9" (UID: "33e31e6c-2296-4e75-9eed-be8579a717b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:25:39.104439 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.104417 6036 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:25:39.104487 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.104436 6036 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33e31e6c-2296-4e75-9eed-be8579a717b9-tmp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:25:39.104540 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.104518 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert" pod "33e31e6c-2296-4e75-9eed-be8579a717b9" (UID: "33e31e6c-2296-4e75-9eed-be8579a717b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:25:39.104592 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.104544 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle" pod "33e31e6c-2296-4e75-9eed-be8579a717b9" (UID: "33e31e6c-2296-4e75-9eed-be8579a717b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:25:39.106317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.106268 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh" pod "33e31e6c-2296-4e75-9eed-be8579a717b9" (UID: "33e31e6c-2296-4e75-9eed-be8579a717b9"). InnerVolumeSpecName "kube-api-access-cnxjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:25:39.106750 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.106728 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config" pod "33e31e6c-2296-4e75-9eed-be8579a717b9" (UID: "33e31e6c-2296-4e75-9eed-be8579a717b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:25:39.106853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.106801 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert" pod "33e31e6c-2296-4e75-9eed-be8579a717b9" (UID: "33e31e6c-2296-4e75-9eed-be8579a717b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:25:39.205753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.205716 6036 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-console-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:25:39.205753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.205747 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnxjh\" (UniqueName: \"kubernetes.io/projected/33e31e6c-2296-4e75-9eed-be8579a717b9-kube-api-access-cnxjh\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:25:39.205753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.205759 6036 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-serving-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:25:39.205985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.205770 6036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-trusted-ca-bundle\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:25:39.205985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.205786 6036 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33e31e6c-2296-4e75-9eed-be8579a717b9-oauth-serving-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:25:39.205985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.205799 6036 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33e31e6c-2296-4e75-9eed-be8579a717b9-console-oauth-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:25:39.700734 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.700699 6036 generic.go:356] "Generic (PLEG): container finished" podID="33e31e6c-2296-4e75-9eed-be8579a717b9" containerID="b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077" exitCode=0 Nov 01 00:25:39.700880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.700749 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585bf7664f-h64rc" Nov 01 00:25:39.700880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.700764 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585bf7664f-h64rc" event={"ID":"33e31e6c-2296-4e75-9eed-be8579a717b9","Type":"ContainerDied","Data":"b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077"} Nov 01 00:25:39.700880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.700805 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585bf7664f-h64rc" event={"ID":"33e31e6c-2296-4e75-9eed-be8579a717b9","Type":"ContainerDied","Data":"ba4f0543a9d2da11d38ed09e342bf56da79f0291f08c95373d2362cb63c40865"} Nov 01 00:25:39.700880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.700823 6036 scope.go:122] "RemoveContainer" containerID="b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077" Nov 01 00:25:39.708894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.708874 6036 scope.go:122] "RemoveContainer" containerID="b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077" Nov 01 00:25:39.709179 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:25:39.709160 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077\": container with ID starting with b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077 not found: ID does not exist" containerID="b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077" Nov 01 00:25:39.709231 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.709188 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077"} err="failed to get container status \"b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077\": rpc error: code = NotFound desc = could not find container \"b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077\": container with ID starting with b0fa0e5c25478ce70d6956831f0c04c5e0fac9696a262d72dca9dbca1c5a3077 not found: ID does not exist" Nov 01 00:25:39.866890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.866855 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585bf7664f-h64rc"] Nov 01 00:25:39.917141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:39.917109 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-585bf7664f-h64rc"] Nov 01 00:25:41.248172 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:25:41.248126 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e31e6c-2296-4e75-9eed-be8579a717b9" path="/var/lib/kubelet/pods/33e31e6c-2296-4e75-9eed-be8579a717b9/volumes" Nov 01 00:25:49.503387 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:25:49.503347 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:26:01.197162 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:01.197116 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:26:01.213586 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:01.213556 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:26:01.800579 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:01.800553 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 00:26:08.687490 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:08.687457 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:08.895217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:08.895177 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:16.416461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.416430 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cfb559466-kms4t"] Nov 01 00:26:16.416851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.416753 6036 cpu_manager.go:414] "containerMap: removing container" podUID="33e31e6c-2296-4e75-9eed-be8579a717b9" containerName="console" Nov 01 00:26:16.416851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.416763 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e31e6c-2296-4e75-9eed-be8579a717b9" containerName="console" Nov 01 00:26:16.416851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.416834 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e31e6c-2296-4e75-9eed-be8579a717b9" containerName="console" Nov 01 00:26:16.421595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.421578 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.434350 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.434322 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-config\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.434467 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.434356 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-oauth-config\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.434467 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.434385 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-oauth-serving-cert\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.434544 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.434497 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-service-ca\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.434544 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.434516 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwlch\" (UniqueName: \"kubernetes.io/projected/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-kube-api-access-xwlch\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.434608 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.434583 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-trusted-ca-bundle\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.434641 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.434621 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-serving-cert\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.434685 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.434646 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-tmp\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.521238 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.521206 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cfb559466-kms4t"] Nov 01 00:26:16.535711 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.535677 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-serving-cert\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.535858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.535714 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-tmp\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.535858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.535743 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-config\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.535858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.535763 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-oauth-config\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.535858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.535848 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-oauth-serving-cert\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.536055 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.535937 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-service-ca\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.536055 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.535971 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwlch\" (UniqueName: \"kubernetes.io/projected/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-kube-api-access-xwlch\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.536055 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.536000 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-trusted-ca-bundle\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.536536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.536510 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-tmp\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.536651 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.536628 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-oauth-serving-cert\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.536717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.536636 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-config\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.536904 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.536878 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-service-ca\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.537274 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.537253 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-trusted-ca-bundle\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.538768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.538745 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-oauth-config\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.539218 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.539201 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-console-serving-cert\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.597783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.597749 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwlch\" (UniqueName: \"kubernetes.io/projected/0d2f77b8-14d6-41e4-bbb3-e01668871ae7-kube-api-access-xwlch\") pod \"console-cfb559466-kms4t\" (UID: \"0d2f77b8-14d6-41e4-bbb3-e01668871ae7\") " pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.707252 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.707169 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:16.732892 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.732872 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:16.935174 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:16.935141 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:17.108141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:17.108049 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cfb559466-kms4t"] Nov 01 00:26:17.110731 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:26:17.110703 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2f77b8_14d6_41e4_bbb3_e01668871ae7.slice/crio-f08db3d233755e86794254c0bc6f6d0b9e80b295dc257393dd6e21c23885018e WatchSource:0}: Error finding container f08db3d233755e86794254c0bc6f6d0b9e80b295dc257393dd6e21c23885018e: Status 404 returned error can't find the container with id f08db3d233755e86794254c0bc6f6d0b9e80b295dc257393dd6e21c23885018e Nov 01 00:26:17.153912 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:17.153889 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:17.366232 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:17.366150 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:17.842550 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:17.842509 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfb559466-kms4t" event={"ID":"0d2f77b8-14d6-41e4-bbb3-e01668871ae7","Type":"ContainerStarted","Data":"0f5fbd3ef9bdca5a1297e585b943b26ca578841d2c8e8f8eab5a9ed0b6c12da0"} Nov 01 00:26:17.842920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:17.842568 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfb559466-kms4t" event={"ID":"0d2f77b8-14d6-41e4-bbb3-e01668871ae7","Type":"ContainerStarted","Data":"f08db3d233755e86794254c0bc6f6d0b9e80b295dc257393dd6e21c23885018e"} Nov 01 00:26:18.013647 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:18.013592 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cfb559466-kms4t" podStartSLOduration=2.013576692 podStartE2EDuration="2.013576692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:26:17.963377835 +0000 UTC m=+122.838598850" watchObservedRunningTime="2026-11-01 00:26:18.013576692 +0000 UTC m=+122.888797703" Nov 01 00:26:19.502728 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:26:19.502689 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:26:20.684850 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:20.684814 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:21.339319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:21.339271 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:21.543715 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:21.543684 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:26.733529 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:26.733492 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:26.733980 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:26.733559 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:26.738425 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:26.738402 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:26.885326 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:26.885275 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cfb559466-kms4t" Nov 01 00:26:26.921571 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:26.921539 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:27.033957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:27.033923 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c4c47bfd-n46cz"] Nov 01 00:26:27.124727 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:27.124692 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:46.558214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:46.558180 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:26:49.503529 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:26:49.503491 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:26:52.054767 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.054703 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66c4c47bfd-n46cz" podUID="365c1b95-a563-4310-82f2-894d776e88cd" containerName="console" containerID="cri-o://a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f" gracePeriod=15 Nov 01 00:26:52.300018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.299992 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:26:52.375724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.375635 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-serving-cert\") pod \"365c1b95-a563-4310-82f2-894d776e88cd\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " Nov 01 00:26:52.375724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.375677 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365c1b95-a563-4310-82f2-894d776e88cd-kube-api-access-8fkfj\" (UniqueName: \"kubernetes.io/projected/365c1b95-a563-4310-82f2-894d776e88cd-kube-api-access-8fkfj\") pod \"365c1b95-a563-4310-82f2-894d776e88cd\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " Nov 01 00:26:52.375724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.375706 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-console-config\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-console-config\") pod \"365c1b95-a563-4310-82f2-894d776e88cd\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " Nov 01 00:26:52.375724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.375727 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-oauth-serving-cert\") pod \"365c1b95-a563-4310-82f2-894d776e88cd\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " Nov 01 00:26:52.376114 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.375767 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-trusted-ca-bundle\") pod \"365c1b95-a563-4310-82f2-894d776e88cd\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " Nov 01 00:26:52.376114 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.375784 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-oauth-config\") pod \"365c1b95-a563-4310-82f2-894d776e88cd\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " Nov 01 00:26:52.376114 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.375802 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365c1b95-a563-4310-82f2-894d776e88cd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/365c1b95-a563-4310-82f2-894d776e88cd-tmp\") pod \"365c1b95-a563-4310-82f2-894d776e88cd\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " Nov 01 00:26:52.376114 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.375961 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-service-ca\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-service-ca\") pod \"365c1b95-a563-4310-82f2-894d776e88cd\" (UID: \"365c1b95-a563-4310-82f2-894d776e88cd\") " Nov 01 00:26:52.376367 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.376067 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-console-config" pod "365c1b95-a563-4310-82f2-894d776e88cd" (UID: "365c1b95-a563-4310-82f2-894d776e88cd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:26:52.376367 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.376176 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365c1b95-a563-4310-82f2-894d776e88cd-tmp" pod "365c1b95-a563-4310-82f2-894d776e88cd" (UID: "365c1b95-a563-4310-82f2-894d776e88cd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:26:52.376367 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.376276 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-trusted-ca-bundle" pod "365c1b95-a563-4310-82f2-894d776e88cd" (UID: "365c1b95-a563-4310-82f2-894d776e88cd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:26:52.376687 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.376439 6036 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-console-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:26:52.376687 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.376459 6036 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/365c1b95-a563-4310-82f2-894d776e88cd-tmp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:26:52.376687 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.376543 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-oauth-serving-cert" pod "365c1b95-a563-4310-82f2-894d776e88cd" (UID: "365c1b95-a563-4310-82f2-894d776e88cd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:26:52.376687 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.376624 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-service-ca" pod "365c1b95-a563-4310-82f2-894d776e88cd" (UID: "365c1b95-a563-4310-82f2-894d776e88cd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:26:52.378059 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.378032 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-serving-cert" pod "365c1b95-a563-4310-82f2-894d776e88cd" (UID: "365c1b95-a563-4310-82f2-894d776e88cd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:26:52.378612 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.378582 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-oauth-config" pod "365c1b95-a563-4310-82f2-894d776e88cd" (UID: "365c1b95-a563-4310-82f2-894d776e88cd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:26:52.378612 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.378601 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365c1b95-a563-4310-82f2-894d776e88cd-kube-api-access-8fkfj" pod "365c1b95-a563-4310-82f2-894d776e88cd" (UID: "365c1b95-a563-4310-82f2-894d776e88cd"). InnerVolumeSpecName "kube-api-access-8fkfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:26:52.476837 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.476797 6036 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-serving-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:26:52.476837 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.476832 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fkfj\" (UniqueName: \"kubernetes.io/projected/365c1b95-a563-4310-82f2-894d776e88cd-kube-api-access-8fkfj\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:26:52.476837 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.476847 6036 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-oauth-serving-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:26:52.477084 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.476861 6036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-trusted-ca-bundle\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:26:52.477084 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.476873 6036 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365c1b95-a563-4310-82f2-894d776e88cd-console-oauth-config\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:26:52.477084 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.476885 6036 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365c1b95-a563-4310-82f2-894d776e88cd-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:26:52.984903 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.984865 6036 generic.go:356] "Generic (PLEG): container finished" podID="365c1b95-a563-4310-82f2-894d776e88cd" containerID="a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f" exitCode=0 Nov 01 00:26:52.985086 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.984915 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c4c47bfd-n46cz" Nov 01 00:26:52.985086 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.984927 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c4c47bfd-n46cz" event={"ID":"365c1b95-a563-4310-82f2-894d776e88cd","Type":"ContainerDied","Data":"a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f"} Nov 01 00:26:52.985086 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.984967 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c4c47bfd-n46cz" event={"ID":"365c1b95-a563-4310-82f2-894d776e88cd","Type":"ContainerDied","Data":"c20ba0721f77e8b98d6a7758cfb222b0e3053c22955722da73b3f51fb44742a4"} Nov 01 00:26:52.985086 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.984987 6036 scope.go:122] "RemoveContainer" containerID="a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f" Nov 01 00:26:52.993814 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.993797 6036 scope.go:122] "RemoveContainer" containerID="a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f" Nov 01 00:26:52.994086 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:26:52.994068 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f\": container with ID starting with a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f not found: ID does not exist" containerID="a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f" Nov 01 00:26:52.994133 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:52.994095 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f"} err="failed to get container status \"a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f\": rpc error: code = NotFound desc = could not find container \"a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f\": container with ID starting with a0dc2fc2ac488e1ebc33a2c102887cc09aaf12fe5fcdf3423852e584faa6604f not found: ID does not exist" Nov 01 00:26:53.150980 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:53.150947 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c4c47bfd-n46cz"] Nov 01 00:26:53.201492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:53.201458 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66c4c47bfd-n46cz"] Nov 01 00:26:53.248690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:26:53.248617 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365c1b95-a563-4310-82f2-894d776e88cd" path="/var/lib/kubelet/pods/365c1b95-a563-4310-82f2-894d776e88cd/volumes" Nov 01 00:27:19.503077 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:27:19.503036 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:27:21.043928 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:27:21.043887 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:27:49.503047 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:27:49.503001 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:28:19.502619 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:28:19.502580 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:28:21.393916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:28:21.393882 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:28:49.503547 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:28:49.503512 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:29:19.503480 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:29:19.503438 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:29:21.723155 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:29:21.723107 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:29:49.502194 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:29:49.502159 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:30:16.342617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:30:16.342581 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:30:16.548938 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:30:16.548910 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:30:19.502531 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:30:19.502496 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:30:22.102450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:30:22.102408 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:30:49.491128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:30:49.491042 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:30:49.503924 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:30:49.503884 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:30:49.698120 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:30:49.698089 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:31:19.503725 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:31:19.503674 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:31:22.430989 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:31:22.430955 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:31:49.502458 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:31:49.502410 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:32:19.502911 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:32:19.502866 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:32:22.786354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:22.786316 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:38.115861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:38.115822 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:38.374592 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:38.374501 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:39.749650 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:39.749621 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:39.964448 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:39.964417 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:41.046210 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:41.046180 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:46.418730 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:46.418697 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:46.637649 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:46.637606 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:47.992422 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:47.992389 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:48.212733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:48.212708 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:48.415535 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:48.415507 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:49.502912 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:32:49.502877 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:32:52.279879 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:52.279849 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:32:55.970411 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:32:55.970363 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:19.503672 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:33:19.503555 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:33:23.152788 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:23.152753 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:34.093572 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:34.093493 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:34.922751 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:34.922717 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96"] Nov 01 00:33:34.923088 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:34.923075 6036 cpu_manager.go:414] "containerMap: removing container" podUID="365c1b95-a563-4310-82f2-894d776e88cd" containerName="console" Nov 01 00:33:34.923141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:34.923089 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365c1b95-a563-4310-82f2-894d776e88cd" containerName="console" Nov 01 00:33:34.923184 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:34.923156 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="365c1b95-a563-4310-82f2-894d776e88cd" containerName="console" Nov 01 00:33:34.926446 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:34.926421 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:34.973335 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:34.973300 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96"] Nov 01 00:33:35.015199 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.015156 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 00:33:35.015426 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.015255 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 00:33:35.015426 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.015270 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 00:33:35.015426 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.015257 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 00:33:35.015426 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.015352 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 00:33:35.015426 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.015403 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-28jl7\"" Nov 01 00:33:35.082143 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.082106 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.082335 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.082150 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7szm\" (UniqueName: \"kubernetes.io/projected/02ca043a-9bab-4924-a21b-857418e73f00-kube-api-access-f7szm\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.082335 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.082257 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.082410 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.082337 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.082410 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.082364 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02ca043a-9bab-4924-a21b-857418e73f00-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.082410 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.082385 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ca043a-9bab-4924-a21b-857418e73f00-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.183564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.183477 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.183564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.183540 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7szm\" (UniqueName: \"kubernetes.io/projected/02ca043a-9bab-4924-a21b-857418e73f00-kube-api-access-f7szm\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.183990 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.183592 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.183990 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.183635 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.183990 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.183667 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02ca043a-9bab-4924-a21b-857418e73f00-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.183990 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.183698 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ca043a-9bab-4924-a21b-857418e73f00-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.184191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.184096 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02ca043a-9bab-4924-a21b-857418e73f00-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.186066 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.186038 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ca043a-9bab-4924-a21b-857418e73f00-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.186249 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.186229 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.186344 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.186328 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.186384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.186338 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02ca043a-9bab-4924-a21b-857418e73f00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.245916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.245883 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7szm\" (UniqueName: \"kubernetes.io/projected/02ca043a-9bab-4924-a21b-857418e73f00-kube-api-access-f7szm\") pod \"opendatahub-operator-controller-manager-7b76c875b9-pxm96\" (UID: \"02ca043a-9bab-4924-a21b-857418e73f00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.541701 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.541674 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:35.681846 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:35.681811 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:36.124605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:36.124581 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96"] Nov 01 00:33:36.127518 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:33:36.127492 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ca043a_9bab_4924_a21b_857418e73f00.slice/crio-72030b16cdeba729da08fc11ef0b9dee66543324caea2542468bf72a18a523b6 WatchSource:0}: Error finding container 72030b16cdeba729da08fc11ef0b9dee66543324caea2542468bf72a18a523b6: Status 404 returned error can't find the container with id 72030b16cdeba729da08fc11ef0b9dee66543324caea2542468bf72a18a523b6 Nov 01 00:33:36.129160 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:36.129145 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:33:36.488011 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:36.487931 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" event={"ID":"02ca043a-9bab-4924-a21b-857418e73f00","Type":"ContainerStarted","Data":"72030b16cdeba729da08fc11ef0b9dee66543324caea2542468bf72a18a523b6"} Nov 01 00:33:37.871423 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:37.871393 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:38.076055 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:38.076014 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:39.846704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:39.846615 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:40.508131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:40.508094 6036 generic.go:356] "Generic (PLEG): container finished" podID="02ca043a-9bab-4924-a21b-857418e73f00" containerID="eea8158e006ede85e31d28fc1e0238d204298d54185cbf77023fe4381e763fd8" exitCode=0 Nov 01 00:33:40.508370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:40.508147 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" event={"ID":"02ca043a-9bab-4924-a21b-857418e73f00","Type":"ContainerDied","Data":"eea8158e006ede85e31d28fc1e0238d204298d54185cbf77023fe4381e763fd8"} Nov 01 00:33:41.514859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:41.514772 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" event={"ID":"02ca043a-9bab-4924-a21b-857418e73f00","Type":"ContainerStarted","Data":"c387471f0e9f117bbac3376f462a7f35bd6bc9313bab95352fad0911143c5565"} Nov 01 00:33:41.515244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:41.514917 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:41.769477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:41.769370 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" podStartSLOduration=3.152273105 podStartE2EDuration="7.769352164s" totalImagesPullingTime="4.162839471s" totalInitContainerRuntime="454.239588ms" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:41.718966961 +0000 UTC m=+566.594188000" watchObservedRunningTime="2026-11-01 00:33:41.769352164 +0000 UTC m=+566.644573178" Nov 01 00:33:43.465474 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:43.465446 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:43.741173 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:43.741083 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:46.266089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:46.266049 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:46.519814 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:46.519722 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:46.757653 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:46.757618 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:48.390325 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.390274 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm"] Nov 01 00:33:48.396816 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.396794 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" Nov 01 00:33:48.407326 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.407300 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9643224-b028-439c-8ca9-a927f72c91a4-tmp\") pod \"jobset-operator-674cd59dc5-bl9mm\" (UID: \"c9643224-b028-439c-8ca9-a927f72c91a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" Nov 01 00:33:48.407429 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.407373 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhxv\" (UniqueName: \"kubernetes.io/projected/c9643224-b028-439c-8ca9-a927f72c91a4-kube-api-access-xbhxv\") pod \"jobset-operator-674cd59dc5-bl9mm\" (UID: \"c9643224-b028-439c-8ca9-a927f72c91a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" Nov 01 00:33:48.501430 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.501392 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 00:33:48.501430 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.501392 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:33:48.501648 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.501489 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-bz9ls\"" Nov 01 00:33:48.502258 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.502240 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm"] Nov 01 00:33:48.508795 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.508764 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9643224-b028-439c-8ca9-a927f72c91a4-tmp\") pod \"jobset-operator-674cd59dc5-bl9mm\" (UID: \"c9643224-b028-439c-8ca9-a927f72c91a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" Nov 01 00:33:48.508921 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.508849 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhxv\" (UniqueName: \"kubernetes.io/projected/c9643224-b028-439c-8ca9-a927f72c91a4-kube-api-access-xbhxv\") pod \"jobset-operator-674cd59dc5-bl9mm\" (UID: \"c9643224-b028-439c-8ca9-a927f72c91a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" Nov 01 00:33:48.509169 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.509151 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9643224-b028-439c-8ca9-a927f72c91a4-tmp\") pod \"jobset-operator-674cd59dc5-bl9mm\" (UID: \"c9643224-b028-439c-8ca9-a927f72c91a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" Nov 01 00:33:48.580083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.580045 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhxv\" (UniqueName: \"kubernetes.io/projected/c9643224-b028-439c-8ca9-a927f72c91a4-kube-api-access-xbhxv\") pod \"jobset-operator-674cd59dc5-bl9mm\" (UID: \"c9643224-b028-439c-8ca9-a927f72c91a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" Nov 01 00:33:48.717857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:48.717767 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" Nov 01 00:33:49.020915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:49.020814 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:49.095897 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:49.095866 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm"] Nov 01 00:33:49.098865 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:33:49.098822 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9643224_b028_439c_8ca9_a927f72c91a4.slice/crio-a35a8836be398e49ffbf1f44def8359c3630319f6d0c8a4faae7f6648b5dcbe6 WatchSource:0}: Error finding container a35a8836be398e49ffbf1f44def8359c3630319f6d0c8a4faae7f6648b5dcbe6: Status 404 returned error can't find the container with id a35a8836be398e49ffbf1f44def8359c3630319f6d0c8a4faae7f6648b5dcbe6 Nov 01 00:33:49.266750 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:49.266707 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:49.503350 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:33:49.503317 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:33:49.549572 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:49.549542 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" event={"ID":"c9643224-b028-439c-8ca9-a927f72c91a4","Type":"ContainerStarted","Data":"a35a8836be398e49ffbf1f44def8359c3630319f6d0c8a4faae7f6648b5dcbe6"} Nov 01 00:33:50.638627 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:50.638581 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:51.340170 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.340120 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:51.792570 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.792540 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh"] Nov 01 00:33:51.799356 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.799326 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" Nov 01 00:33:51.835422 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.835385 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56dd2777-c103-48bd-965b-98584249a286-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qcdkh\" (UID: \"56dd2777-c103-48bd-965b-98584249a286\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" Nov 01 00:33:51.835594 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.835562 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4k6\" (UniqueName: \"kubernetes.io/projected/56dd2777-c103-48bd-965b-98584249a286-kube-api-access-6k4k6\") pod \"cert-manager-cainjector-5c57bbd4fc-qcdkh\" (UID: \"56dd2777-c103-48bd-965b-98584249a286\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" Nov 01 00:33:51.866692 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.866658 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qhtjd\"" Nov 01 00:33:51.871752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.871694 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:51.912981 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.912943 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 00:33:51.912981 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.912945 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 00:33:51.913965 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.913936 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh"] Nov 01 00:33:51.936565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.936534 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56dd2777-c103-48bd-965b-98584249a286-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qcdkh\" (UID: \"56dd2777-c103-48bd-965b-98584249a286\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" Nov 01 00:33:51.936757 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.936624 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4k6\" (UniqueName: \"kubernetes.io/projected/56dd2777-c103-48bd-965b-98584249a286-kube-api-access-6k4k6\") pod \"cert-manager-cainjector-5c57bbd4fc-qcdkh\" (UID: \"56dd2777-c103-48bd-965b-98584249a286\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" Nov 01 00:33:51.998236 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.998205 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56dd2777-c103-48bd-965b-98584249a286-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qcdkh\" (UID: \"56dd2777-c103-48bd-965b-98584249a286\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" Nov 01 00:33:51.998420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:51.998249 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4k6\" (UniqueName: \"kubernetes.io/projected/56dd2777-c103-48bd-965b-98584249a286-kube-api-access-6k4k6\") pod \"cert-manager-cainjector-5c57bbd4fc-qcdkh\" (UID: \"56dd2777-c103-48bd-965b-98584249a286\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" Nov 01 00:33:52.097738 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:52.097655 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:52.115916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:52.115887 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" Nov 01 00:33:52.495676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:52.495600 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh"] Nov 01 00:33:52.499845 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:33:52.499812 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56dd2777_c103_48bd_965b_98584249a286.slice/crio-cb78db0e2a7e0079896ec5d6f849756fab45a72f016f7233e63cf63e20226583 WatchSource:0}: Error finding container cb78db0e2a7e0079896ec5d6f849756fab45a72f016f7233e63cf63e20226583: Status 404 returned error can't find the container with id cb78db0e2a7e0079896ec5d6f849756fab45a72f016f7233e63cf63e20226583 Nov 01 00:33:52.522391 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:52.522362 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-pxm96" Nov 01 00:33:52.564141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:52.564108 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" event={"ID":"56dd2777-c103-48bd-965b-98584249a286","Type":"ContainerStarted","Data":"cb78db0e2a7e0079896ec5d6f849756fab45a72f016f7233e63cf63e20226583"} Nov 01 00:33:53.570179 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:53.570142 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" event={"ID":"c9643224-b028-439c-8ca9-a927f72c91a4","Type":"ContainerStarted","Data":"4ad4446324e2872b48727fcaa233ed7c7b50347ffa0cd7c0c29a4b7bdc8191f2"} Nov 01 00:33:53.687145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:53.687083 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bl9mm" podStartSLOduration=1.820396502 podStartE2EDuration="5.687066647s" totalImagesPullingTime="3.866670145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:53.686857091 +0000 UTC m=+578.562078108" watchObservedRunningTime="2026-11-01 00:33:53.687066647 +0000 UTC m=+578.562287663" Nov 01 00:33:56.587868 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:56.587832 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" event={"ID":"56dd2777-c103-48bd-965b-98584249a286","Type":"ContainerStarted","Data":"e95482a62c99c6207c6a285ddf0265cb1d95bc662dd79d7fb82c555a37176901"} Nov 01 00:33:56.704085 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:56.703988 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qcdkh" podStartSLOduration=1.751517497 podStartE2EDuration="5.703974859s" totalImagesPullingTime="3.952457362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:56.703279813 +0000 UTC m=+581.578500843" watchObservedRunningTime="2026-11-01 00:33:56.703974859 +0000 UTC m=+581.579195874" Nov 01 00:33:57.857237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:57.857205 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:58.099444 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:58.099409 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:58.312459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:58.312429 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:58.740503 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:58.740411 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:33:59.318734 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.318703 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-dl7qj"] Nov 01 00:33:59.320909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.320891 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dl7qj" Nov 01 00:33:59.404638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.404603 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0f7f5c5-0320-4402-b493-f4128415976f-bound-sa-token\") pod \"cert-manager-597b95cb84-dl7qj\" (UID: \"b0f7f5c5-0320-4402-b493-f4128415976f\") " pod="cert-manager/cert-manager-597b95cb84-dl7qj" Nov 01 00:33:59.404809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.404653 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g865n\" (UniqueName: \"kubernetes.io/projected/b0f7f5c5-0320-4402-b493-f4128415976f-kube-api-access-g865n\") pod \"cert-manager-597b95cb84-dl7qj\" (UID: \"b0f7f5c5-0320-4402-b493-f4128415976f\") " pod="cert-manager/cert-manager-597b95cb84-dl7qj" Nov 01 00:33:59.424189 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.424167 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-x4zx9\"" Nov 01 00:33:59.425152 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.425134 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dl7qj"] Nov 01 00:33:59.505921 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.505886 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0f7f5c5-0320-4402-b493-f4128415976f-bound-sa-token\") pod \"cert-manager-597b95cb84-dl7qj\" (UID: \"b0f7f5c5-0320-4402-b493-f4128415976f\") " pod="cert-manager/cert-manager-597b95cb84-dl7qj" Nov 01 00:33:59.505921 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.505931 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g865n\" (UniqueName: \"kubernetes.io/projected/b0f7f5c5-0320-4402-b493-f4128415976f-kube-api-access-g865n\") pod \"cert-manager-597b95cb84-dl7qj\" (UID: \"b0f7f5c5-0320-4402-b493-f4128415976f\") " pod="cert-manager/cert-manager-597b95cb84-dl7qj" Nov 01 00:33:59.567487 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.567452 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0f7f5c5-0320-4402-b493-f4128415976f-bound-sa-token\") pod \"cert-manager-597b95cb84-dl7qj\" (UID: \"b0f7f5c5-0320-4402-b493-f4128415976f\") " pod="cert-manager/cert-manager-597b95cb84-dl7qj" Nov 01 00:33:59.567636 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.567559 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g865n\" (UniqueName: \"kubernetes.io/projected/b0f7f5c5-0320-4402-b493-f4128415976f-kube-api-access-g865n\") pod \"cert-manager-597b95cb84-dl7qj\" (UID: \"b0f7f5c5-0320-4402-b493-f4128415976f\") " pod="cert-manager/cert-manager-597b95cb84-dl7qj" Nov 01 00:33:59.630633 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:33:59.630552 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dl7qj" Nov 01 00:34:00.013386 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:00.011982 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dl7qj"] Nov 01 00:34:00.015565 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:34:00.015517 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0f7f5c5_0320_4402_b493_f4128415976f.slice/crio-da65ba021ec40e58c4e06e94c21e702cadd0ff486bef15fd5dbac8504c5acf02 WatchSource:0}: Error finding container da65ba021ec40e58c4e06e94c21e702cadd0ff486bef15fd5dbac8504c5acf02: Status 404 returned error can't find the container with id da65ba021ec40e58c4e06e94c21e702cadd0ff486bef15fd5dbac8504c5acf02 Nov 01 00:34:00.121649 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:00.121612 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:00.341138 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:00.341101 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:00.606119 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:00.606029 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dl7qj" event={"ID":"b0f7f5c5-0320-4402-b493-f4128415976f","Type":"ContainerStarted","Data":"957caf0466336a32e83dc57c51ef01b9faf58c850ed5e53f5e7646193e348498"} Nov 01 00:34:00.606119 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:00.606077 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dl7qj" event={"ID":"b0f7f5c5-0320-4402-b493-f4128415976f","Type":"ContainerStarted","Data":"da65ba021ec40e58c4e06e94c21e702cadd0ff486bef15fd5dbac8504c5acf02"} Nov 01 00:34:00.729916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:00.729850 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-dl7qj" podStartSLOduration=1.729830961 podStartE2EDuration="1.729830961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:00.729326781 +0000 UTC m=+585.604547797" watchObservedRunningTime="2026-11-01 00:34:00.729830961 +0000 UTC m=+585.605051977" Nov 01 00:34:00.857790 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:00.857654 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:01.492912 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:01.492874 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:02.051162 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:02.051129 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:03.076534 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:03.076497 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:04.511095 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:04.511062 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:04.724100 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:04.724057 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:06.846923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:06.846890 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:07.085607 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:07.085576 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:09.841857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:09.841825 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:10.049688 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:10.049658 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:10.259976 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:10.259900 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:10.483222 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:10.483189 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:11.897748 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:11.897713 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:12.115171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:12.115137 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:12.571963 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:12.571927 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:13.753651 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.753607 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:13.755558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.755535 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr"] Nov 01 00:34:13.757922 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.757905 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.853353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.853315 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-tls-certs\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.853536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.853371 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvnq\" (UniqueName: \"kubernetes.io/projected/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-kube-api-access-lrvnq\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.853576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.853529 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-tmp\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.858824 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.858786 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 00:34:13.862391 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.862364 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9sfnp\"" Nov 01 00:34:13.862516 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.862483 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 00:34:13.862583 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.862564 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 00:34:13.863541 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.863524 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr"] Nov 01 00:34:13.911451 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.911421 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 00:34:13.954101 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.954068 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-tmp\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.954266 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.954116 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-tls-certs\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.954266 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.954157 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvnq\" (UniqueName: \"kubernetes.io/projected/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-kube-api-access-lrvnq\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.956464 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.956436 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-tmp\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.956662 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.956644 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-tls-certs\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:13.965651 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:13.965626 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:14.018529 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:14.018458 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvnq\" (UniqueName: \"kubernetes.io/projected/ec4613c5-e6a3-461f-96ac-a3a2d17e99ae-kube-api-access-lrvnq\") pod \"kube-auth-proxy-69c59cdf59-gbhrr\" (UID: \"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae\") " pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:14.069428 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:14.069393 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" Nov 01 00:34:14.166918 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:14.166886 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:14.440698 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:14.440663 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr"] Nov 01 00:34:14.443682 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:34:14.443652 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4613c5_e6a3_461f_96ac_a3a2d17e99ae.slice/crio-bebbde92a5b84ca861593e21456ed743099be03195769400ab4b1eaeee4dc9ee WatchSource:0}: Error finding container bebbde92a5b84ca861593e21456ed743099be03195769400ab4b1eaeee4dc9ee: Status 404 returned error can't find the container with id bebbde92a5b84ca861593e21456ed743099be03195769400ab4b1eaeee4dc9ee Nov 01 00:34:14.533469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:14.533441 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:14.667347 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:14.667279 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" event={"ID":"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae","Type":"ContainerStarted","Data":"bebbde92a5b84ca861593e21456ed743099be03195769400ab4b1eaeee4dc9ee"} Nov 01 00:34:14.742190 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:14.742098 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:18.261739 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:18.261698 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:18.470565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:18.470528 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:18.544718 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:18.544696 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 00:34:18.689425 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:18.689389 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" event={"ID":"ec4613c5-e6a3-461f-96ac-a3a2d17e99ae","Type":"ContainerStarted","Data":"ed7a6174f4115f50e01e98d24c67f184ded89f3d9794c1a489bfb72a920122aa"} Nov 01 00:34:18.806621 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:18.806516 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69c59cdf59-gbhrr" podStartSLOduration=1.7603498279999998 podStartE2EDuration="5.806500367s" totalImagesPullingTime="4.046150539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:18.805761183 +0000 UTC m=+603.680982201" watchObservedRunningTime="2026-11-01 00:34:18.806500367 +0000 UTC m=+603.681721432" Nov 01 00:34:19.502812 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:34:19.502775 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:34:21.477991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:21.477948 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:23.542386 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:23.542351 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:24.286537 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:24.286499 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:24.539341 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:24.539224 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:24.745777 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:24.745742 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:24.978988 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:24.978903 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:25.683221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.683183 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq"] Nov 01 00:34:25.687574 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.687552 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.791304 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.791263 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-g4wf6\"" Nov 01 00:34:25.791709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.791343 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 01 00:34:25.791709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.791371 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 01 00:34:25.791709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.791374 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 01 00:34:25.792435 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.792416 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq"] Nov 01 00:34:25.874470 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.874432 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e3e3da0-57f1-4eb0-bc27-ae322f793193-cert\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.874653 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.874518 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e3e3da0-57f1-4eb0-bc27-ae322f793193-metrics-certs\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.874653 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.874561 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4e3e3da0-57f1-4eb0-bc27-ae322f793193-manager-config\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.874736 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.874674 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz75f\" (UniqueName: \"kubernetes.io/projected/4e3e3da0-57f1-4eb0-bc27-ae322f793193-kube-api-access-mz75f\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.975535 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.975438 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz75f\" (UniqueName: \"kubernetes.io/projected/4e3e3da0-57f1-4eb0-bc27-ae322f793193-kube-api-access-mz75f\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.975703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.975534 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e3e3da0-57f1-4eb0-bc27-ae322f793193-cert\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.975703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.975569 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e3e3da0-57f1-4eb0-bc27-ae322f793193-metrics-certs\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.975703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.975622 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4e3e3da0-57f1-4eb0-bc27-ae322f793193-manager-config\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.976406 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.976374 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4e3e3da0-57f1-4eb0-bc27-ae322f793193-manager-config\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.978144 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.978113 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e3e3da0-57f1-4eb0-bc27-ae322f793193-metrics-certs\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:25.978267 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:25.978160 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e3e3da0-57f1-4eb0-bc27-ae322f793193-cert\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:26.037432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:26.037395 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz75f\" (UniqueName: \"kubernetes.io/projected/4e3e3da0-57f1-4eb0-bc27-ae322f793193-kube-api-access-mz75f\") pod \"jobset-controller-manager-785d6989b9-xfstq\" (UID: \"4e3e3da0-57f1-4eb0-bc27-ae322f793193\") " pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:26.298713 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:26.298673 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:26.671325 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:26.671226 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq"] Nov 01 00:34:26.674119 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:34:26.674085 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3e3da0_57f1_4eb0_bc27_ae322f793193.slice/crio-870ee1dd64407601af7bede54597f919d736396cda72c9bcbbb2e51f87d7dfd1 WatchSource:0}: Error finding container 870ee1dd64407601af7bede54597f919d736396cda72c9bcbbb2e51f87d7dfd1: Status 404 returned error can't find the container with id 870ee1dd64407601af7bede54597f919d736396cda72c9bcbbb2e51f87d7dfd1 Nov 01 00:34:26.731017 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:26.730978 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" event={"ID":"4e3e3da0-57f1-4eb0-bc27-ae322f793193","Type":"ContainerStarted","Data":"870ee1dd64407601af7bede54597f919d736396cda72c9bcbbb2e51f87d7dfd1"} Nov 01 00:34:29.534574 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:29.534509 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:29.742315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:29.742263 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:31.754093 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:31.754052 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" event={"ID":"4e3e3da0-57f1-4eb0-bc27-ae322f793193","Type":"ContainerStarted","Data":"90f48c0de9e7ddc456fd8cd687cc7c557ff42cb67266bb51c43f67a1aad4add9"} Nov 01 00:34:31.754624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:31.754196 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:31.874794 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:31.874741 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" podStartSLOduration=2.276613735 podStartE2EDuration="6.874726019s" totalImagesPullingTime="4.598112284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:31.87393575 +0000 UTC m=+616.749156765" watchObservedRunningTime="2026-11-01 00:34:31.874726019 +0000 UTC m=+616.749947110" Nov 01 00:34:33.161918 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:33.161881 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:33.384991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:33.384956 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:34.860008 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:34.859979 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:35.227554 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:35.227436 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:35.432279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:35.432247 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:36.604444 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:36.604411 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:36.810018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:36.809987 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:38.305024 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:38.304990 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:38.516724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:38.516675 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:38.729302 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:38.729209 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:38.935124 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:38.935089 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:41.073341 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.073306 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm"] Nov 01 00:34:41.075604 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.075588 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.105268 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.105238 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6dd774a7-0574-4eb9-9c7f-a7cff549f713-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.105448 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.105310 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6dd774a7-0574-4eb9-9c7f-a7cff549f713-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.105448 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.105410 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htldl\" (UniqueName: \"kubernetes.io/projected/6dd774a7-0574-4eb9-9c7f-a7cff549f713-kube-api-access-htldl\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.105448 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.105445 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dd774a7-0574-4eb9-9c7f-a7cff549f713-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.128836 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.128806 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 00:34:41.179406 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.179369 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-xqplh\"" Nov 01 00:34:41.179609 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.179375 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 00:34:41.206426 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.206393 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6dd774a7-0574-4eb9-9c7f-a7cff549f713-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.206589 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.206462 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htldl\" (UniqueName: \"kubernetes.io/projected/6dd774a7-0574-4eb9-9c7f-a7cff549f713-kube-api-access-htldl\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.206589 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.206502 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dd774a7-0574-4eb9-9c7f-a7cff549f713-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.206589 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.206556 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6dd774a7-0574-4eb9-9c7f-a7cff549f713-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.206826 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.206808 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6dd774a7-0574-4eb9-9c7f-a7cff549f713-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.206886 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.206837 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dd774a7-0574-4eb9-9c7f-a7cff549f713-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.281210 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.281175 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 01 00:34:41.282273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.282250 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm"] Nov 01 00:34:41.283988 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.283970 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htldl\" (UniqueName: \"kubernetes.io/projected/6dd774a7-0574-4eb9-9c7f-a7cff549f713-kube-api-access-htldl\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.287730 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.287706 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6dd774a7-0574-4eb9-9c7f-a7cff549f713-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p5lhm\" (UID: \"6dd774a7-0574-4eb9-9c7f-a7cff549f713\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.386607 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.386527 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:41.812339 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:41.812122 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm"] Nov 01 00:34:41.815406 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:34:41.815373 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd774a7_0574_4eb9_9c7f_a7cff549f713.slice/crio-f4d66d378cdd3b0977439f8032e1aa1abd95070af467759ad90e6f0428b87c73 WatchSource:0}: Error finding container f4d66d378cdd3b0977439f8032e1aa1abd95070af467759ad90e6f0428b87c73: Status 404 returned error can't find the container with id f4d66d378cdd3b0977439f8032e1aa1abd95070af467759ad90e6f0428b87c73 Nov 01 00:34:42.768845 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:42.768814 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-785d6989b9-xfstq" Nov 01 00:34:42.804111 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:42.804075 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" event={"ID":"6dd774a7-0574-4eb9-9c7f-a7cff549f713","Type":"ContainerStarted","Data":"f4d66d378cdd3b0977439f8032e1aa1abd95070af467759ad90e6f0428b87c73"} Nov 01 00:34:43.989950 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:43.989914 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:44.199323 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:44.199248 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:34:45.825029 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:45.824949 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" event={"ID":"6dd774a7-0574-4eb9-9c7f-a7cff549f713","Type":"ContainerStarted","Data":"ff68768160b4fbf8a87f20388e6437a957c876213d3337ceb54e4942d4bf661d"} Nov 01 00:34:45.825411 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:45.825050 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:34:45.995450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:34:45.995392 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" podStartSLOduration=1.241966774 podStartE2EDuration="4.995378667s" totalImagesPullingTime="3.753411893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:45.994789487 +0000 UTC m=+630.870010502" watchObservedRunningTime="2026-11-01 00:34:45.995378667 +0000 UTC m=+630.870599725" Nov 01 00:34:49.503838 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:34:49.503802 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:35:00.294346 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:00.294253 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:00.513385 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:00.513351 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:00.966760 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:00.966728 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:06.831387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:06.831359 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p5lhm" Nov 01 00:35:19.507112 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:35:19.507072 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:35:21.739225 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:21.739186 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:21.956892 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:21.956852 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:22.191596 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:22.191559 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:23.869650 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:23.869613 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:28.342261 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:28.342222 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:28.550549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:28.550506 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:31.453463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:31.453423 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:31.667401 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:31.667372 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:48.644145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:48.644114 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:48.852647 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:48.852611 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:49.083016 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.082975 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:49.297740 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.297707 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:49.505195 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:35:49.505113 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:35:49.510701 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.510670 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:49.671324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.671267 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-z622g"] Nov 01 00:35:49.678426 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.678398 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:49.726924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.726886 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:49.732680 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.732648 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-z622g"] Nov 01 00:35:49.770664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.770581 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff1a9a2-2384-4fc2-8487-7f1387461d60-tls-certs\") pod \"model-serving-api-75bd89fdfc-z622g\" (UID: \"eff1a9a2-2384-4fc2-8487-7f1387461d60\") " pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:49.770664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.770657 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9f9j\" (UniqueName: \"kubernetes.io/projected/eff1a9a2-2384-4fc2-8487-7f1387461d60-kube-api-access-x9f9j\") pod \"model-serving-api-75bd89fdfc-z622g\" (UID: \"eff1a9a2-2384-4fc2-8487-7f1387461d60\") " pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:49.782126 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.782096 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-njps8\"" Nov 01 00:35:49.782303 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.782160 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 00:35:49.871769 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.871726 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff1a9a2-2384-4fc2-8487-7f1387461d60-tls-certs\") pod \"model-serving-api-75bd89fdfc-z622g\" (UID: \"eff1a9a2-2384-4fc2-8487-7f1387461d60\") " pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:49.871957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.871809 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9f9j\" (UniqueName: \"kubernetes.io/projected/eff1a9a2-2384-4fc2-8487-7f1387461d60-kube-api-access-x9f9j\") pod \"model-serving-api-75bd89fdfc-z622g\" (UID: \"eff1a9a2-2384-4fc2-8487-7f1387461d60\") " pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:49.874417 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.874390 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff1a9a2-2384-4fc2-8487-7f1387461d60-tls-certs\") pod \"model-serving-api-75bd89fdfc-z622g\" (UID: \"eff1a9a2-2384-4fc2-8487-7f1387461d60\") " pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:49.934088 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.934052 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9f9j\" (UniqueName: \"kubernetes.io/projected/eff1a9a2-2384-4fc2-8487-7f1387461d60-kube-api-access-x9f9j\") pod \"model-serving-api-75bd89fdfc-z622g\" (UID: \"eff1a9a2-2384-4fc2-8487-7f1387461d60\") " pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:49.990465 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:49.990420 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:50.361560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:50.361486 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-z622g"] Nov 01 00:35:50.364724 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:35:50.364690 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff1a9a2_2384_4fc2_8487_7f1387461d60.slice/crio-92cf690008ff158399414849c2725aebdc154b64aa217536ce8d5856af525047 WatchSource:0}: Error finding container 92cf690008ff158399414849c2725aebdc154b64aa217536ce8d5856af525047: Status 404 returned error can't find the container with id 92cf690008ff158399414849c2725aebdc154b64aa217536ce8d5856af525047 Nov 01 00:35:50.878052 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:50.878021 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:51.106441 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:51.106408 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-z622g" event={"ID":"eff1a9a2-2384-4fc2-8487-7f1387461d60","Type":"ContainerStarted","Data":"92cf690008ff158399414849c2725aebdc154b64aa217536ce8d5856af525047"} Nov 01 00:35:53.116589 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:53.116556 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-z622g" event={"ID":"eff1a9a2-2384-4fc2-8487-7f1387461d60","Type":"ContainerStarted","Data":"d5a0d3f94af5b42e12739996c975cdba5266862d7a0460982edfcc76102c03af"} Nov 01 00:35:53.117022 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:53.116684 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:35:53.235640 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:53.235586 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-z622g" podStartSLOduration=1.932926704 podStartE2EDuration="4.235570345s" totalImagesPullingTime="2.302643641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:53.234559905 +0000 UTC m=+698.109780922" watchObservedRunningTime="2026-11-01 00:35:53.235570345 +0000 UTC m=+698.110791415" Nov 01 00:35:55.904583 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:55.904550 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:56.110632 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:56.110602 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:56.348270 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:56.348241 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:56.568728 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:56.568690 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:56.809754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:56.809723 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:57.021807 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:57.021773 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:35:57.337801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:35:57.337771 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:00.931705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:00.931666 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:01.140551 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:01.140522 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:03.066089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:03.066041 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:03.272479 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:03.272440 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:03.478989 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:03.478905 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:03.691984 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:03.691952 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:04.125339 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:04.125278 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-z622g" Nov 01 00:36:15.722608 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:15.722577 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:15.971612 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:15.971579 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:16.132798 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.132760 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w"] Nov 01 00:36:16.136257 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.136235 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.140391 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.140360 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-maas-api-tls\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.140528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.140393 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4276cd-4570-4993-afb7-a3b23c388b39-openshift-service-ca\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.140581 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.140563 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppnz\" (UniqueName: \"kubernetes.io/projected/9c4276cd-4570-4993-afb7-a3b23c388b39-kube-api-access-6ppnz\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.140616 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.140603 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-metrics-certs\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.185450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.185419 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:16.240122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240092 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 00:36:16.240122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240106 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 00:36:16.240122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240092 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 00:36:16.240122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240114 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8jq2v\"" Nov 01 00:36:16.240447 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240147 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 00:36:16.240934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240915 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-maas-api-tls\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.240977 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240942 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4276cd-4570-4993-afb7-a3b23c388b39-openshift-service-ca\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.241026 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240983 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppnz\" (UniqueName: \"kubernetes.io/projected/9c4276cd-4570-4993-afb7-a3b23c388b39-kube-api-access-6ppnz\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.241026 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.240999 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-metrics-certs\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.241494 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.241474 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w"] Nov 01 00:36:16.241717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.241694 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4276cd-4570-4993-afb7-a3b23c388b39-openshift-service-ca\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.243721 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.243699 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-metrics-certs\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.243721 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.243715 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-maas-api-tls\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.303012 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.302969 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppnz\" (UniqueName: \"kubernetes.io/projected/9c4276cd-4570-4993-afb7-a3b23c388b39-kube-api-access-6ppnz\") pod \"maas-api-5c7d8966cd-nxm5w\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.449162 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.449070 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:16.827792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:16.827769 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w"] Nov 01 00:36:16.830414 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:36:16.830381 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4276cd_4570_4993_afb7_a3b23c388b39.slice/crio-ee2d4a8dea6131e16564c5446d030e8e36f903ba607e08f19c8fd1bd778296a5 WatchSource:0}: Error finding container ee2d4a8dea6131e16564c5446d030e8e36f903ba607e08f19c8fd1bd778296a5: Status 404 returned error can't find the container with id ee2d4a8dea6131e16564c5446d030e8e36f903ba607e08f19c8fd1bd778296a5 Nov 01 00:36:17.225112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:17.225028 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" event={"ID":"9c4276cd-4570-4993-afb7-a3b23c388b39","Type":"ContainerStarted","Data":"ee2d4a8dea6131e16564c5446d030e8e36f903ba607e08f19c8fd1bd778296a5"} Nov 01 00:36:19.509349 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:36:19.509306 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:36:20.242700 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:20.242666 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" event={"ID":"9c4276cd-4570-4993-afb7-a3b23c388b39","Type":"ContainerStarted","Data":"34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059"} Nov 01 00:36:20.243050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:20.243027 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:20.362703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:20.362649 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" podStartSLOduration=1.509129843 podStartE2EDuration="4.362635424s" totalImagesPullingTime="2.853505581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:36:20.362198563 +0000 UTC m=+725.237419578" watchObservedRunningTime="2026-11-01 00:36:20.362635424 +0000 UTC m=+725.237856438" Nov 01 00:36:24.185634 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:24.185552 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:36:26.252878 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:36:26.252851 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 00:36:49.503590 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:36:49.503549 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:37:19.504158 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:37:19.504111 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:37:24.552024 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:24.551980 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:37:34.502128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:34.502096 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:37:34.710382 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:34.710350 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:37:49.503709 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:37:49.503667 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:37:54.299742 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:54.299646 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:37:57.814160 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.814124 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn"] Nov 01 00:37:57.817643 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.817623 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.871214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.871183 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-s6ps6\"" Nov 01 00:37:57.891736 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.891705 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.891736 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.891739 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dhn\" (UniqueName: \"kubernetes.io/projected/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-kube-api-access-q2dhn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.891920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.891759 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.891920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.891891 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.891920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.891914 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.892018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.891935 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.921280 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.921252 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 00:37:57.921576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.921557 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 00:37:57.921676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.921582 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 00:37:57.922696 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.922675 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn"] Nov 01 00:37:57.982857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.982825 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:37:57.993488 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993460 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.993577 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993499 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.993614 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993587 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.993652 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993618 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dhn\" (UniqueName: \"kubernetes.io/projected/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-kube-api-access-q2dhn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.993652 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993644 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.993731 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993716 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.993941 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993920 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.994027 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993942 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.994027 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.993985 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.996145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.996118 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:57.996384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:57.996362 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:58.055361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:58.055327 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dhn\" (UniqueName: \"kubernetes.io/projected/e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04-kube-api-access-q2dhn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7f5bn\" (UID: \"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:58.130357 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:58.130251 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:37:58.503726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:58.503672 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn"] Nov 01 00:37:58.506047 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:37:58.506017 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f6ff66_a7bf_4aef_9cb1_8b2b86eddb04.slice/crio-571ff6b9e59a4cafa13f6f043e5159ec1ca7ab6f45f88ee19236d7fd8f9588e0 WatchSource:0}: Error finding container 571ff6b9e59a4cafa13f6f043e5159ec1ca7ab6f45f88ee19236d7fd8f9588e0: Status 404 returned error can't find the container with id 571ff6b9e59a4cafa13f6f043e5159ec1ca7ab6f45f88ee19236d7fd8f9588e0 Nov 01 00:37:58.687453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:37:58.687418 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" event={"ID":"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04","Type":"ContainerStarted","Data":"571ff6b9e59a4cafa13f6f043e5159ec1ca7ab6f45f88ee19236d7fd8f9588e0"} Nov 01 00:38:00.698572 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:00.698544 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" event={"ID":"e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04","Type":"ContainerStarted","Data":"55a510a46faa5d6a217cb368a1c1906980576584161a733c7ffd53a8f8a64f9e"} Nov 01 00:38:00.818081 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:00.818027 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" podStartSLOduration=2.096684587 podStartE2EDuration="3.818014098s" totalImagesPullingTime="1.721329511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:00.816876399 +0000 UTC m=+825.692097416" watchObservedRunningTime="2026-11-01 00:38:00.818014098 +0000 UTC m=+825.693235113" Nov 01 00:38:01.131119 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:01.131079 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:38:05.141147 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:05.141105 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:08.131226 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:08.131193 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:38:08.143402 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:08.143377 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:38:08.747310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:08.747264 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" Nov 01 00:38:09.407660 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:09.407629 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:12.562330 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.562280 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tsnn6"] Nov 01 00:38:12.566177 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.566155 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.631794 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.631764 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.631954 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.631910 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.631954 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.631943 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwp6\" (UniqueName: \"kubernetes.io/projected/53c3efef-0317-4a09-9edb-3462eac4eb96-kube-api-access-mdwp6\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.632036 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.631962 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-home\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.632036 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.631983 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.632128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.632053 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c3efef-0317-4a09-9edb-3462eac4eb96-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.669599 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.669564 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 01 00:38:12.670865 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.670844 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tsnn6"] Nov 01 00:38:12.733006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.732974 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-home\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.733200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.733014 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.733200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.733042 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c3efef-0317-4a09-9edb-3462eac4eb96-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.733200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.733078 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.733200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.733173 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.733200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.733200 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwp6\" (UniqueName: \"kubernetes.io/projected/53c3efef-0317-4a09-9edb-3462eac4eb96-kube-api-access-mdwp6\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.733515 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.733488 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.733591 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.733540 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-home\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.733591 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.733555 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.735679 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.735651 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53c3efef-0317-4a09-9edb-3462eac4eb96-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.735872 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.735852 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c3efef-0317-4a09-9edb-3462eac4eb96-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.742434 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.742407 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:12.795006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.794967 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwp6\" (UniqueName: \"kubernetes.io/projected/53c3efef-0317-4a09-9edb-3462eac4eb96-kube-api-access-mdwp6\") pod \"echo-simulator-kserve-64b6f8f96c-tsnn6\" (UID: \"53c3efef-0317-4a09-9edb-3462eac4eb96\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:12.878496 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:12.878392 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:13.250431 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:13.250361 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tsnn6"] Nov 01 00:38:13.253531 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:38:13.253498 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c3efef_0317_4a09_9edb_3462eac4eb96.slice/crio-b9a9e846fd49464d17a1883d90aaf50eecf5a5199169a62e0456f7eb5d371661 WatchSource:0}: Error finding container b9a9e846fd49464d17a1883d90aaf50eecf5a5199169a62e0456f7eb5d371661: Status 404 returned error can't find the container with id b9a9e846fd49464d17a1883d90aaf50eecf5a5199169a62e0456f7eb5d371661 Nov 01 00:38:13.759544 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:13.759509 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" event={"ID":"53c3efef-0317-4a09-9edb-3462eac4eb96","Type":"ContainerStarted","Data":"b9a9e846fd49464d17a1883d90aaf50eecf5a5199169a62e0456f7eb5d371661"} Nov 01 00:38:16.444325 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:16.444208 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:16.777255 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:16.777166 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" event={"ID":"53c3efef-0317-4a09-9edb-3462eac4eb96","Type":"ContainerStarted","Data":"a0b7a04d35bcfa5d73b2563db17c480b3e7d4e4d256d5c5abc7745d4ff036b5f"} Nov 01 00:38:16.878875 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:16.878843 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:16.898042 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:16.897990 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" podStartSLOduration=2.068414955 podStartE2EDuration="4.897975476s" totalImagesPullingTime="2.829560521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:16.897082088 +0000 UTC m=+841.772303106" watchObservedRunningTime="2026-11-01 00:38:16.897975476 +0000 UTC m=+841.773196490" Nov 01 00:38:19.503973 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:38:19.503867 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:38:20.416242 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.416206 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx"] Nov 01 00:38:20.419770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.419753 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.511581 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.511545 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.511991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.511592 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01758a46-b41c-4ac1-b4b7-6baf086384e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.511991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.511621 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.511991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.511719 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5z7z\" (UniqueName: \"kubernetes.io/projected/01758a46-b41c-4ac1-b4b7-6baf086384e4-kube-api-access-m5z7z\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.511991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.511767 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.511991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.511786 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.523201 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.523178 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 01 00:38:20.524462 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.524424 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx"] Nov 01 00:38:20.595675 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.595641 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:20.612298 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612255 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.612484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612326 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01758a46-b41c-4ac1-b4b7-6baf086384e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.612484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612361 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.612484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612401 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5z7z\" (UniqueName: \"kubernetes.io/projected/01758a46-b41c-4ac1-b4b7-6baf086384e4-kube-api-access-m5z7z\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.612484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612431 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.612484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612459 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.612776 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612684 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.612874 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612846 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.613017 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.612858 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.614836 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.614818 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01758a46-b41c-4ac1-b4b7-6baf086384e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.615009 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.614991 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01758a46-b41c-4ac1-b4b7-6baf086384e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.674704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.674629 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5z7z\" (UniqueName: \"kubernetes.io/projected/01758a46-b41c-4ac1-b4b7-6baf086384e4-kube-api-access-m5z7z\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx\" (UID: \"01758a46-b41c-4ac1-b4b7-6baf086384e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:20.735044 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:20.734983 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:21.111450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:21.111425 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx"] Nov 01 00:38:21.113899 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:38:21.113874 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01758a46_b41c_4ac1_b4b7_6baf086384e4.slice/crio-e5869b249739f9e38bfa7543aa59f8f0a39a44cd2f22a73e860e519c776b52f4 WatchSource:0}: Error finding container e5869b249739f9e38bfa7543aa59f8f0a39a44cd2f22a73e860e519c776b52f4: Status 404 returned error can't find the container with id e5869b249739f9e38bfa7543aa59f8f0a39a44cd2f22a73e860e519c776b52f4 Nov 01 00:38:21.804626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:21.804593 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" event={"ID":"01758a46-b41c-4ac1-b4b7-6baf086384e4","Type":"ContainerStarted","Data":"e5869b249739f9e38bfa7543aa59f8f0a39a44cd2f22a73e860e519c776b52f4"} Nov 01 00:38:22.810439 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:22.810401 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" event={"ID":"01758a46-b41c-4ac1-b4b7-6baf086384e4","Type":"ContainerStarted","Data":"8b0fff3a6cff93c28a552524ceaf043744537dad2f1df46a156d1f959b4a8485"} Nov 01 00:38:22.879300 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:22.879261 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:22.891722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:22.891696 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:22.928190 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:22.928136 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podStartSLOduration=2.143937128 podStartE2EDuration="2.928119769s" totalImagesPullingTime="784.182641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:22.927726053 +0000 UTC m=+847.802947070" watchObservedRunningTime="2026-11-01 00:38:22.928119769 +0000 UTC m=+847.803340785" Nov 01 00:38:23.735347 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:23.735307 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:23.825980 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:23.825952 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tsnn6" Nov 01 00:38:24.307529 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.307500 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv"] Nov 01 00:38:24.311398 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.311377 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.347908 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.347872 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.347908 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.347903 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.348260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.348235 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.348358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.348330 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgd2p\" (UniqueName: \"kubernetes.io/projected/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-kube-api-access-fgd2p\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.348405 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.348364 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.348482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.348469 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.364611 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.364584 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 00:38:24.416344 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.416310 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv"] Nov 01 00:38:24.449766 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.449727 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.449970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.449782 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.449970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.449808 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.450099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.449969 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.450099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.450042 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgd2p\" (UniqueName: \"kubernetes.io/projected/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-kube-api-access-fgd2p\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.450099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.450061 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.450244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.450119 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.450244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.450190 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.450458 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.450427 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.452328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.452309 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.452504 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.452487 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.463656 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.463628 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:24.513506 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.513477 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgd2p\" (UniqueName: \"kubernetes.io/projected/44a07f20-d1e3-40cb-aa2d-66ed878c22d5-kube-api-access-fgd2p\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv\" (UID: \"44a07f20-d1e3-40cb-aa2d-66ed878c22d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.623528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.623434 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:24.900789 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.900709 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:24.995444 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:24.995421 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv"] Nov 01 00:38:24.998050 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:38:24.998016 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a07f20_d1e3_40cb_aa2d_66ed878c22d5.slice/crio-548db45d756fcbb423f4ad0504425ab79d2e870ea71b8871665430abf0ea66d6 WatchSource:0}: Error finding container 548db45d756fcbb423f4ad0504425ab79d2e870ea71b8871665430abf0ea66d6: Status 404 returned error can't find the container with id 548db45d756fcbb423f4ad0504425ab79d2e870ea71b8871665430abf0ea66d6 Nov 01 00:38:25.826623 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:25.826593 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" event={"ID":"44a07f20-d1e3-40cb-aa2d-66ed878c22d5","Type":"ContainerStarted","Data":"548db45d756fcbb423f4ad0504425ab79d2e870ea71b8871665430abf0ea66d6"} Nov 01 00:38:26.832237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:26.832202 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" event={"ID":"44a07f20-d1e3-40cb-aa2d-66ed878c22d5","Type":"ContainerStarted","Data":"8a51321906e55d7cc8dd7ab0bb62bed9e4090397f026315035e11f307c1b589e"} Nov 01 00:38:26.952184 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:26.952128 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" podStartSLOduration=2.175180851 podStartE2EDuration="2.952113691s" totalImagesPullingTime="776.93284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:26.950838196 +0000 UTC m=+851.826059216" watchObservedRunningTime="2026-11-01 00:38:26.952113691 +0000 UTC m=+851.827334706" Nov 01 00:38:27.623869 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:27.623832 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:30.735585 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:30.735546 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:30.753765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:30.753731 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:30.862152 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:30.862120 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:38:31.774046 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:31.774008 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:34.624359 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:34.624315 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:34.637038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:34.637013 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:34.881065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:34.880976 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv" Nov 01 00:38:35.629677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.629643 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl"] Nov 01 00:38:35.635752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.635728 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.688552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.688519 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 00:38:35.740642 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.740615 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl"] Nov 01 00:38:35.761512 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761479 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkgr\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-kube-api-access-khkgr\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.761667 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761524 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.761667 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761545 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6313eca7-5cd0-472f-be31-2860c6b47e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.761667 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761627 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.761667 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761642 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.761801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761673 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.761801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761695 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.761866 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761794 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.761904 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.761894 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.776879 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.776855 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srhv5\"" Nov 01 00:38:35.863277 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863250 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863336 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkgr\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-kube-api-access-khkgr\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863373 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863401 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6313eca7-5cd0-472f-be31-2860c6b47e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863435 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863459 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863495 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863523 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863559 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.863981 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.863956 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.864041 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.864008 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.864099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.864057 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.864204 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.864153 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.864204 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.864220 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6313eca7-5cd0-472f-be31-2860c6b47e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.865872 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.865844 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.865994 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.865917 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.925033 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.924959 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.925174 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.925098 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkgr\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-kube-api-access-khkgr\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzxfl\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:35.949524 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:35.949495 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:36.375872 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:36.375850 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl"] Nov 01 00:38:36.378248 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:38:36.378217 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6313eca7_5cd0_472f_be31_2860c6b47e65.slice/crio-307bb8392b815749e01b2bf143ef78b305d79a1406c6829ce5395ed72080b175 WatchSource:0}: Error finding container 307bb8392b815749e01b2bf143ef78b305d79a1406c6829ce5395ed72080b175: Status 404 returned error can't find the container with id 307bb8392b815749e01b2bf143ef78b305d79a1406c6829ce5395ed72080b175 Nov 01 00:38:36.380358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:36.380343 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:38:36.880719 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:36.880684 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" event={"ID":"6313eca7-5cd0-472f-be31-2860c6b47e65","Type":"ContainerStarted","Data":"307bb8392b815749e01b2bf143ef78b305d79a1406c6829ce5395ed72080b175"} Nov 01 00:38:40.342216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:40.342185 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:38:40.342482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:40.342252 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:38:40.342482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:40.342308 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:38:40.902983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:40.902952 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" event={"ID":"6313eca7-5cd0-472f-be31-2860c6b47e65","Type":"ContainerStarted","Data":"cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426"} Nov 01 00:38:40.969792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:40.969760 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:41.139820 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:41.139765 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" podStartSLOduration=2.178973745 podStartE2EDuration="6.139749479s" totalImagesPullingTime="3.960775734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:41.088831088 +0000 UTC m=+865.964052108" watchObservedRunningTime="2026-11-01 00:38:41.139749479 +0000 UTC m=+866.014970493" Nov 01 00:38:41.969319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:41.968422 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzxfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:38:41.969319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:41.968506 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 00:38:42.952324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:42.951708 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzxfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:38:42.952324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:42.951769 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 00:38:43.965744 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:43.965697 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzxfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:38:43.966411 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:43.966371 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 00:38:44.964313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:44.962706 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzxfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:38:44.964313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:44.962781 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 00:38:45.960310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:45.959681 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzxfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:38:45.960310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:45.959742 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 00:38:46.950681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:46.950640 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzxfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:38:46.950895 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:46.950692 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 00:38:48.522347 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.522313 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-c87zz"] Nov 01 00:38:48.529237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.529209 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:48.571891 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.571851 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:48.583970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.583928 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-c87zz"] Nov 01 00:38:48.602600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.602569 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:48.602957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.602929 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:48.603949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.603930 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:38:48.633262 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.633225 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 00:38:48.633262 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.633232 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 00:38:48.633485 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.633274 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-6p7zv\"" Nov 01 00:38:48.711405 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.711371 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077b80e1-f2a9-49cb-a32c-799656e96e60-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-c87zz\" (UID: \"077b80e1-f2a9-49cb-a32c-799656e96e60\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:48.711676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.711650 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9gq\" (UniqueName: \"kubernetes.io/projected/077b80e1-f2a9-49cb-a32c-799656e96e60-kube-api-access-nr9gq\") pod \"ai-gateway-controller-7d6c696b49-c87zz\" (UID: \"077b80e1-f2a9-49cb-a32c-799656e96e60\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:48.812676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.812649 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9gq\" (UniqueName: \"kubernetes.io/projected/077b80e1-f2a9-49cb-a32c-799656e96e60-kube-api-access-nr9gq\") pod \"ai-gateway-controller-7d6c696b49-c87zz\" (UID: \"077b80e1-f2a9-49cb-a32c-799656e96e60\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:48.812838 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.812697 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077b80e1-f2a9-49cb-a32c-799656e96e60-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-c87zz\" (UID: \"077b80e1-f2a9-49cb-a32c-799656e96e60\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:48.815505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.815481 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077b80e1-f2a9-49cb-a32c-799656e96e60-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-c87zz\" (UID: \"077b80e1-f2a9-49cb-a32c-799656e96e60\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:48.874635 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:48.874593 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9gq\" (UniqueName: \"kubernetes.io/projected/077b80e1-f2a9-49cb-a32c-799656e96e60-kube-api-access-nr9gq\") pod \"ai-gateway-controller-7d6c696b49-c87zz\" (UID: \"077b80e1-f2a9-49cb-a32c-799656e96e60\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:49.144639 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:49.144542 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:49.503797 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:38:49.503714 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:38:49.519597 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:49.519567 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-c87zz"] Nov 01 00:38:49.523056 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:38:49.523027 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077b80e1_f2a9_49cb_a32c_799656e96e60.slice/crio-88269f46e0c7d2ad085033abde1d4227af1fc0ebafc65455b51af7efad8d011f WatchSource:0}: Error finding container 88269f46e0c7d2ad085033abde1d4227af1fc0ebafc65455b51af7efad8d011f: Status 404 returned error can't find the container with id 88269f46e0c7d2ad085033abde1d4227af1fc0ebafc65455b51af7efad8d011f Nov 01 00:38:49.948393 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:49.948349 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" event={"ID":"077b80e1-f2a9-49cb-a32c-799656e96e60","Type":"ContainerStarted","Data":"88269f46e0c7d2ad085033abde1d4227af1fc0ebafc65455b51af7efad8d011f"} Nov 01 00:38:50.743068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:50.743040 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:55.980787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:55.980744 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" event={"ID":"077b80e1-f2a9-49cb-a32c-799656e96e60","Type":"ContainerStarted","Data":"676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e"} Nov 01 00:38:55.981474 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:55.980824 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:38:56.099082 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:56.099029 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" podStartSLOduration=2.472639269 podStartE2EDuration="8.099006905s" totalImagesPullingTime="5.626367636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:38:56.098005025 +0000 UTC m=+880.973226041" watchObservedRunningTime="2026-11-01 00:38:56.099006905 +0000 UTC m=+880.974227971" Nov 01 00:38:58.015783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:58.015749 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:38:58.216365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:38:58.216327 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:06.987224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:06.987190 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:39:07.361823 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.361791 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7784f485d-ftt6z"] Nov 01 00:39:07.365393 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.365374 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:07.424715 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.424683 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7784f485d-ftt6z"] Nov 01 00:39:07.492704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.492668 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f221f59-060d-4828-b652-da453b2558c4-metrics-certs\") pod \"ai-gateway-controller-7784f485d-ftt6z\" (UID: \"3f221f59-060d-4828-b652-da453b2558c4\") " pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:07.492903 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.492718 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pt6\" (UniqueName: \"kubernetes.io/projected/3f221f59-060d-4828-b652-da453b2558c4-kube-api-access-n7pt6\") pod \"ai-gateway-controller-7784f485d-ftt6z\" (UID: \"3f221f59-060d-4828-b652-da453b2558c4\") " pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:07.593947 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.593901 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pt6\" (UniqueName: \"kubernetes.io/projected/3f221f59-060d-4828-b652-da453b2558c4-kube-api-access-n7pt6\") pod \"ai-gateway-controller-7784f485d-ftt6z\" (UID: \"3f221f59-060d-4828-b652-da453b2558c4\") " pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:07.594150 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.594060 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f221f59-060d-4828-b652-da453b2558c4-metrics-certs\") pod \"ai-gateway-controller-7784f485d-ftt6z\" (UID: \"3f221f59-060d-4828-b652-da453b2558c4\") " pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:07.596662 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.596636 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f221f59-060d-4828-b652-da453b2558c4-metrics-certs\") pod \"ai-gateway-controller-7784f485d-ftt6z\" (UID: \"3f221f59-060d-4828-b652-da453b2558c4\") " pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:07.656074 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.656002 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pt6\" (UniqueName: \"kubernetes.io/projected/3f221f59-060d-4828-b652-da453b2558c4-kube-api-access-n7pt6\") pod \"ai-gateway-controller-7784f485d-ftt6z\" (UID: \"3f221f59-060d-4828-b652-da453b2558c4\") " pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:07.678043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:07.678009 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:08.050526 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:08.050410 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7784f485d-ftt6z"] Nov 01 00:39:08.053392 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:39:08.053353 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f221f59_060d_4828_b652_da453b2558c4.slice/crio-426bc4e89f3e30882de4594b9ab41a4d0ce3f48606cb3eb22ca086b63418c95a WatchSource:0}: Error finding container 426bc4e89f3e30882de4594b9ab41a4d0ce3f48606cb3eb22ca086b63418c95a: Status 404 returned error can't find the container with id 426bc4e89f3e30882de4594b9ab41a4d0ce3f48606cb3eb22ca086b63418c95a Nov 01 00:39:09.049512 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:09.049478 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" event={"ID":"3f221f59-060d-4828-b652-da453b2558c4","Type":"ContainerStarted","Data":"6729f7e6d7bf7f3636358f4b39326142b77adb2ba6f7e3264272af73ff0302b9"} Nov 01 00:39:09.049700 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:09.049528 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" event={"ID":"3f221f59-060d-4828-b652-da453b2558c4","Type":"ContainerStarted","Data":"426bc4e89f3e30882de4594b9ab41a4d0ce3f48606cb3eb22ca086b63418c95a"} Nov 01 00:39:09.049700 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:09.049623 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:09.167787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:09.167691 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" podStartSLOduration=1.335809499 podStartE2EDuration="2.167676922s" totalImagesPullingTime="831.867423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:39:09.166860166 +0000 UTC m=+894.042081182" watchObservedRunningTime="2026-11-01 00:39:09.167676922 +0000 UTC m=+894.042897937" Nov 01 00:39:19.504104 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:39:19.504065 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:39:20.056274 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.056246 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7784f485d-ftt6z" Nov 01 00:39:20.200713 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.200683 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-c87zz"] Nov 01 00:39:20.200916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.200875 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" podUID="077b80e1-f2a9-49cb-a32c-799656e96e60" containerName="manager" containerID="cri-o://676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e" gracePeriod=30 Nov 01 00:39:20.451728 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.451704 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:39:20.623382 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.623263 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/077b80e1-f2a9-49cb-a32c-799656e96e60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/077b80e1-f2a9-49cb-a32c-799656e96e60-metrics-certs\") pod \"077b80e1-f2a9-49cb-a32c-799656e96e60\" (UID: \"077b80e1-f2a9-49cb-a32c-799656e96e60\") " Nov 01 00:39:20.623824 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.623405 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077b80e1-f2a9-49cb-a32c-799656e96e60-kube-api-access-nr9gq\" (UniqueName: \"kubernetes.io/projected/077b80e1-f2a9-49cb-a32c-799656e96e60-kube-api-access-nr9gq\") pod \"077b80e1-f2a9-49cb-a32c-799656e96e60\" (UID: \"077b80e1-f2a9-49cb-a32c-799656e96e60\") " Nov 01 00:39:20.625702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.625670 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077b80e1-f2a9-49cb-a32c-799656e96e60-kube-api-access-nr9gq" pod "077b80e1-f2a9-49cb-a32c-799656e96e60" (UID: "077b80e1-f2a9-49cb-a32c-799656e96e60"). InnerVolumeSpecName "kube-api-access-nr9gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:39:20.625702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.625682 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077b80e1-f2a9-49cb-a32c-799656e96e60-metrics-certs" pod "077b80e1-f2a9-49cb-a32c-799656e96e60" (UID: "077b80e1-f2a9-49cb-a32c-799656e96e60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:39:20.724119 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.724076 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr9gq\" (UniqueName: \"kubernetes.io/projected/077b80e1-f2a9-49cb-a32c-799656e96e60-kube-api-access-nr9gq\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:39:20.724119 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:20.724110 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077b80e1-f2a9-49cb-a32c-799656e96e60-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:39:21.105244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.105208 6036 generic.go:356] "Generic (PLEG): container finished" podID="077b80e1-f2a9-49cb-a32c-799656e96e60" containerID="676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e" exitCode=0 Nov 01 00:39:21.105484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.105259 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" Nov 01 00:39:21.105484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.105314 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" event={"ID":"077b80e1-f2a9-49cb-a32c-799656e96e60","Type":"ContainerDied","Data":"676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e"} Nov 01 00:39:21.105484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.105374 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-c87zz" event={"ID":"077b80e1-f2a9-49cb-a32c-799656e96e60","Type":"ContainerDied","Data":"88269f46e0c7d2ad085033abde1d4227af1fc0ebafc65455b51af7efad8d011f"} Nov 01 00:39:21.105484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.105402 6036 scope.go:122] "RemoveContainer" containerID="676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e" Nov 01 00:39:21.114815 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.114797 6036 scope.go:122] "RemoveContainer" containerID="676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e" Nov 01 00:39:21.115081 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:39:21.115062 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e\": container with ID starting with 676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e not found: ID does not exist" containerID="676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e" Nov 01 00:39:21.115138 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.115094 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e"} err="failed to get container status \"676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e\": rpc error: code = NotFound desc = could not find container \"676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e\": container with ID starting with 676cd2454961b76928aa2c3c9ec50fcf4af115bfc989edd5396c4b82c6d47e8e not found: ID does not exist" Nov 01 00:39:21.270884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.270850 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-c87zz"] Nov 01 00:39:21.275824 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.275798 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-c87zz"] Nov 01 00:39:21.905558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:21.905524 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:22.404706 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:22.404672 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:23.249138 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:23.249103 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077b80e1-f2a9-49cb-a32c-799656e96e60" path="/var/lib/kubelet/pods/077b80e1-f2a9-49cb-a32c-799656e96e60/volumes" Nov 01 00:39:24.026074 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:24.025993 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:24.249275 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:24.249237 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:25.241267 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:25.241240 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:25.523923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:25.523843 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:25.751089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:25.751054 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:47.801459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:47.801420 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:48.009924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:48.009883 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:49.503450 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:39:49.503412 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:39:57.067191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:57.067132 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:39:57.276891 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:39:57.276859 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:00.145865 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:00.145836 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:00.354940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:00.354905 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:00.806823 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:00.806778 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:00.969919 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:00.969888 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq"] Nov 01 00:40:00.970414 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:00.970382 6036 cpu_manager.go:414] "containerMap: removing container" podUID="077b80e1-f2a9-49cb-a32c-799656e96e60" containerName="manager" Nov 01 00:40:00.970414 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:00.970402 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077b80e1-f2a9-49cb-a32c-799656e96e60" containerName="manager" Nov 01 00:40:00.970656 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:00.970527 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="077b80e1-f2a9-49cb-a32c-799656e96e60" containerName="manager" Nov 01 00:40:00.973608 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:00.973593 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.030105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.030073 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-948e22-openshift-default-dockercfg-2wgzk\"" Nov 01 00:40:01.078819 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.078725 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq"] Nov 01 00:40:01.104998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.104966 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:01.125187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125154 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.125187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125187 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.125449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125217 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.125449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125239 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.125564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125441 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.125564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125487 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.125564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125514 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.125695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125605 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.125695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.125640 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2css\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-kube-api-access-g2css\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.226995 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.226953 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.227536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227007 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.227536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227047 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.227536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227193 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.227536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227231 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.227536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227249 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.227536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227316 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.227536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227344 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2css\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-kube-api-access-g2css\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.227536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227462 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.228145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227728 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.228145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227753 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.228145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227794 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.228145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.227821 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.228458 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.228433 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.229671 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.229651 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.229770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.229751 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.288691 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.288643 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk"] Nov 01 00:40:01.289827 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.289801 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2css\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-kube-api-access-g2css\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.290046 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.290023 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.293747 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.293724 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.327523 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.327485 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:01.347244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.347165 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-22d7b2-openshift-default-dockercfg-lhnwj\"" Nov 01 00:40:01.353812 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.353785 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk"] Nov 01 00:40:01.428918 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.428884 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.429117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.428939 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.429117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.428956 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.429117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.429078 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.429117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.429107 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bht4z\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-kube-api-access-bht4z\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.429363 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.429137 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.429363 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.429243 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5da06e-9f90-4586-b65e-fa6173da4969-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.429363 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.429324 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.429520 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.429406 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.530949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.530911 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531176 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.530988 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531176 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531049 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531176 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531166 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531402 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531212 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531402 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531237 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bht4z\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-kube-api-access-bht4z\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531402 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531311 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531402 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531361 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5da06e-9f90-4586-b65e-fa6173da4969-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531402 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531470 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531561 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531728 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531606 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.531832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.531793 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.532075 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.532057 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5da06e-9f90-4586-b65e-fa6173da4969-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.533340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.533323 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.533934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.533913 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.546316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.546270 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:01.586597 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.586566 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:01.594567 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.594527 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.594723 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.594578 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bht4z\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-kube-api-access-bht4z\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.617545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.617516 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:01.754348 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.754317 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:01.945588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.945504 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82"] Nov 01 00:40:01.950491 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.950468 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:01.997222 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:01.997190 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:02.003881 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.003853 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5a9c19-openshift-default-dockercfg-vrnmb\"" Nov 01 00:40:02.010130 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.010102 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82"] Nov 01 00:40:02.013498 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.013461 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq"] Nov 01 00:40:02.051345 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:02.051317 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4a2c30_4b8f_4d6e_90f8_8569f072d908.slice/crio-932c8a101da903f09af7032e351cba23926de2298a05191b59072a3372193c7c WatchSource:0}: Error finding container 932c8a101da903f09af7032e351cba23926de2298a05191b59072a3372193c7c: Status 404 returned error can't find the container with id 932c8a101da903f09af7032e351cba23926de2298a05191b59072a3372193c7c Nov 01 00:40:02.053543 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.053505 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:02.053650 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.053590 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:02.053650 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.053635 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:02.070882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.070858 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk"] Nov 01 00:40:02.073320 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:02.073276 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5da06e_9f90_4586_b65e_fa6173da4969.slice/crio-a05974e50e51153b6be03fae3513751bcc05db36c46b12f5db1ee42af96c769f WatchSource:0}: Error finding container a05974e50e51153b6be03fae3513751bcc05db36c46b12f5db1ee42af96c769f: Status 404 returned error can't find the container with id a05974e50e51153b6be03fae3513751bcc05db36c46b12f5db1ee42af96c769f Nov 01 00:40:02.075456 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.075421 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:02.075538 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.075489 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:02.075538 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.075522 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:02.139648 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.139608 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.139648 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.139666 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2bc\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-kube-api-access-fw2bc\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.139849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.139692 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a0e24d-d19e-4750-9423-b6f61f034983-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.139919 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.139839 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a0e24d-d19e-4750-9423-b6f61f034983-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.139919 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.139883 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.140016 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.139922 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.140092 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.140067 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.140148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.140124 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.140222 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.140206 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.205199 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.205098 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:02.241306 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241242 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241337 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241386 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241451 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241557 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241597 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2bc\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-kube-api-access-fw2bc\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241632 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a0e24d-d19e-4750-9423-b6f61f034983-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241664 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a0e24d-d19e-4750-9423-b6f61f034983-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241676 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.241752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241697 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.242171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241676 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.242171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.241813 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.242362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.242340 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.242918 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.242600 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a0e24d-d19e-4750-9423-b6f61f034983-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.244687 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.244661 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.244937 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.244919 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a0e24d-d19e-4750-9423-b6f61f034983-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.299231 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.299196 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" event={"ID":"5e4a2c30-4b8f-4d6e-90f8-8569f072d908","Type":"ContainerStarted","Data":"73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091"} Nov 01 00:40:02.299460 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.299262 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" event={"ID":"5e4a2c30-4b8f-4d6e-90f8-8569f072d908","Type":"ContainerStarted","Data":"932c8a101da903f09af7032e351cba23926de2298a05191b59072a3372193c7c"} Nov 01 00:40:02.301474 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.301447 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" event={"ID":"3e5da06e-9f90-4586-b65e-fa6173da4969","Type":"ContainerStarted","Data":"f3c250fde8915e6f0cee6e25a76ccab2a463ea18acbf5a23915e9a4e873772fd"} Nov 01 00:40:02.301646 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.301502 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" event={"ID":"3e5da06e-9f90-4586-b65e-fa6173da4969","Type":"ContainerStarted","Data":"a05974e50e51153b6be03fae3513751bcc05db36c46b12f5db1ee42af96c769f"} Nov 01 00:40:02.306749 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.306710 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.306980 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.306954 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2bc\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-kube-api-access-fw2bc\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.421880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.421814 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" podStartSLOduration=2.421793713 podStartE2EDuration="2.421793713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:02.421669793 +0000 UTC m=+947.296890808" watchObservedRunningTime="2026-11-01 00:40:02.421793713 +0000 UTC m=+947.297014730" Nov 01 00:40:02.468775 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.468652 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:02.548168 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.548116 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" podStartSLOduration=1.5481019599999999 podStartE2EDuration="1.54810196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:02.546896133 +0000 UTC m=+947.422117149" watchObservedRunningTime="2026-11-01 00:40:02.54810196 +0000 UTC m=+947.423322975" Nov 01 00:40:02.564193 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.564138 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:02.588489 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.588462 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:02.590161 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.590132 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4"] Nov 01 00:40:02.594068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.594048 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.619502 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.618636 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:02.655477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.655445 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4"] Nov 01 00:40:02.659358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.659316 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-43a0cf-openshift-default-dockercfg-9jwvj\"" Nov 01 00:40:02.681765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.681732 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:02.748497 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748408 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.748497 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748472 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.748717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748593 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.748717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748633 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.748717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748686 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8rg\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-kube-api-access-lt8rg\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.748843 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748773 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.748843 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748817 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.748953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748899 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.748953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.748945 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850265 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850331 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850363 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850410 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850438 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850457 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850478 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8rg\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-kube-api-access-lt8rg\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850536 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850576 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850719 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.850987 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.850861 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.851056 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.851000 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.851114 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.851057 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.851261 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.851242 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.853169 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.853148 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.853420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.853404 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.889143 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.889111 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:02.912403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.912366 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.912529 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.912474 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8rg\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-kube-api-access-lt8rg\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:02.949315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.949263 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82"] Nov 01 00:40:02.951089 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:02.951062 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a0e24d_d19e_4750_9423_b6f61f034983.slice/crio-03791406567d65aa95ba906cc441c63cdc2b622eac3ec7fd2b2bca5ac894c91a WatchSource:0}: Error finding container 03791406567d65aa95ba906cc441c63cdc2b622eac3ec7fd2b2bca5ac894c91a: Status 404 returned error can't find the container with id 03791406567d65aa95ba906cc441c63cdc2b622eac3ec7fd2b2bca5ac894c91a Nov 01 00:40:02.954088 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.954057 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:02.954161 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.954134 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:02.954202 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:02.954161 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:03.204900 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.204832 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:03.207815 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.207792 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:03.308498 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.308451 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" event={"ID":"26a0e24d-d19e-4750-9423-b6f61f034983","Type":"ContainerStarted","Data":"17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c"} Nov 01 00:40:03.308843 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.308527 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" event={"ID":"26a0e24d-d19e-4750-9423-b6f61f034983","Type":"ContainerStarted","Data":"03791406567d65aa95ba906cc441c63cdc2b622eac3ec7fd2b2bca5ac894c91a"} Nov 01 00:40:03.432306 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.432225 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" podStartSLOduration=2.43220601 podStartE2EDuration="2.43220601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:03.431259135 +0000 UTC m=+948.306480151" watchObservedRunningTime="2026-11-01 00:40:03.43220601 +0000 UTC m=+948.307427028" Nov 01 00:40:03.564577 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.564541 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:03.588984 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.588956 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4"] Nov 01 00:40:03.590865 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:03.590837 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c9ed19_e38a_4bf4_859b_9dde6bf7901e.slice/crio-82662597c7205b761b4be59393c0a250282ad55a317e75b36cfd5d16a680b3c1 WatchSource:0}: Error finding container 82662597c7205b761b4be59393c0a250282ad55a317e75b36cfd5d16a680b3c1: Status 404 returned error can't find the container with id 82662597c7205b761b4be59393c0a250282ad55a317e75b36cfd5d16a680b3c1 Nov 01 00:40:03.593123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.593078 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:03.593822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.593791 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:03.593920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.593863 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:03.593920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.593907 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:03.623409 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:03.623385 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:04.316178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:04.316073 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" event={"ID":"75c9ed19-e38a-4bf4-859b-9dde6bf7901e","Type":"ContainerStarted","Data":"4c92376e8ea155a66c9de70f76d169079d6f8d5d4605fa260f76a9d69789dcab"} Nov 01 00:40:04.316178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:04.316141 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" event={"ID":"75c9ed19-e38a-4bf4-859b-9dde6bf7901e","Type":"ContainerStarted","Data":"82662597c7205b761b4be59393c0a250282ad55a317e75b36cfd5d16a680b3c1"} Nov 01 00:40:04.316764 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:04.316483 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:04.316764 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:04.316523 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:04.317520 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:04.317495 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:04.317725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:04.317711 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:40:04.442809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:04.442753 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podStartSLOduration=2.442739442 podStartE2EDuration="2.442739442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:04.441676975 +0000 UTC m=+949.316898002" watchObservedRunningTime="2026-11-01 00:40:04.442739442 +0000 UTC m=+949.317960456" Nov 01 00:40:04.571366 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:04.571212 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:05.208455 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:05.208420 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:05.213337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:05.213312 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:05.321797 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:05.321767 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:05.321797 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:05.321822 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:05.322972 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:05.322950 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:05.323125 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:05.323049 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:05.689632 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:05.689596 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:05.906356 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:05.906327 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:06.322861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:06.322823 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:06.535877 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:06.535848 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:06.948700 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:06.948667 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:07.157336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:07.157301 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:07.742509 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:07.742477 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:07.971195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:07.971164 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:08.386914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:08.386878 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:08.601997 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:08.601964 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:08.812473 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:08.812441 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:09.031248 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:09.031214 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:09.250343 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:09.250245 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:11.282633 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:11.282598 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:11.488624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:11.488568 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:12.229332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:12.229299 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:12.442739 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:12.442705 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:12.902757 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:12.902722 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:13.117024 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:13.116993 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:16.123051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.123007 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:16.325161 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.325128 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:16.473162 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.473078 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz"] Nov 01 00:40:16.478320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.478269 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:16.535205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.535172 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-22d7b2\"" Nov 01 00:40:16.581964 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.581930 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk76l\" (UniqueName: \"kubernetes.io/projected/675d3e25-4ab9-46be-878a-d049d7e50e58-kube-api-access-vk76l\") pod \"payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz\" (UID: \"675d3e25-4ab9-46be-878a-d049d7e50e58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:16.582135 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.582115 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/675d3e25-4ab9-46be-878a-d049d7e50e58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz\" (UID: \"675d3e25-4ab9-46be-878a-d049d7e50e58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:16.585638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.585618 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-22d7b2-dockercfg-phdkv\"" Nov 01 00:40:16.587336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.587311 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz"] Nov 01 00:40:16.637981 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.637941 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6"] Nov 01 00:40:16.641967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.641941 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:16.683612 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.683580 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751ff7e8-5d91-4386-8882-c7d75d4e668f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6\" (UID: \"751ff7e8-5d91-4386-8882-c7d75d4e668f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:16.683792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.683639 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/675d3e25-4ab9-46be-878a-d049d7e50e58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz\" (UID: \"675d3e25-4ab9-46be-878a-d049d7e50e58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:16.683792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.683752 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk76l\" (UniqueName: \"kubernetes.io/projected/675d3e25-4ab9-46be-878a-d049d7e50e58-kube-api-access-vk76l\") pod \"payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz\" (UID: \"675d3e25-4ab9-46be-878a-d049d7e50e58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:16.683913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.683789 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkp7w\" (UniqueName: \"kubernetes.io/projected/751ff7e8-5d91-4386-8882-c7d75d4e668f-kube-api-access-zkp7w\") pod \"payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6\" (UID: \"751ff7e8-5d91-4386-8882-c7d75d4e668f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:16.684267 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.684246 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/675d3e25-4ab9-46be-878a-d049d7e50e58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz\" (UID: \"675d3e25-4ab9-46be-878a-d049d7e50e58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:16.686963 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.686933 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6"] Nov 01 00:40:16.749109 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.749033 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk76l\" (UniqueName: \"kubernetes.io/projected/675d3e25-4ab9-46be-878a-d049d7e50e58-kube-api-access-vk76l\") pod \"payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz\" (UID: \"675d3e25-4ab9-46be-878a-d049d7e50e58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:16.784659 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.784621 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751ff7e8-5d91-4386-8882-c7d75d4e668f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6\" (UID: \"751ff7e8-5d91-4386-8882-c7d75d4e668f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:16.784841 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.784711 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkp7w\" (UniqueName: \"kubernetes.io/projected/751ff7e8-5d91-4386-8882-c7d75d4e668f-kube-api-access-zkp7w\") pod \"payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6\" (UID: \"751ff7e8-5d91-4386-8882-c7d75d4e668f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:16.785439 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.785400 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751ff7e8-5d91-4386-8882-c7d75d4e668f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6\" (UID: \"751ff7e8-5d91-4386-8882-c7d75d4e668f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:16.790972 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.790951 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:16.847398 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.847372 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkp7w\" (UniqueName: \"kubernetes.io/projected/751ff7e8-5d91-4386-8882-c7d75d4e668f-kube-api-access-zkp7w\") pod \"payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6\" (UID: \"751ff7e8-5d91-4386-8882-c7d75d4e668f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:16.954645 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:16.954610 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:17.175464 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.167259 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz"] Nov 01 00:40:17.327743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.327716 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6"] Nov 01 00:40:17.329958 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:17.329933 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751ff7e8_5d91_4386_8882_c7d75d4e668f.slice/crio-076e9783534a6bf30e9a7a1ef3ffa91324a5441f37d1069075c181faa68609f7 WatchSource:0}: Error finding container 076e9783534a6bf30e9a7a1ef3ffa91324a5441f37d1069075c181faa68609f7: Status 404 returned error can't find the container with id 076e9783534a6bf30e9a7a1ef3ffa91324a5441f37d1069075c181faa68609f7 Nov 01 00:40:17.351171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.351143 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7"] Nov 01 00:40:17.356991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.356970 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.383065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.383029 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" event={"ID":"751ff7e8-5d91-4386-8882-c7d75d4e668f","Type":"ContainerStarted","Data":"076e9783534a6bf30e9a7a1ef3ffa91324a5441f37d1069075c181faa68609f7"} Nov 01 00:40:17.384140 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.384119 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" event={"ID":"675d3e25-4ab9-46be-878a-d049d7e50e58","Type":"ContainerStarted","Data":"9581f662a75b908e532768c9c175e41f3ebf06c33d01a54704ac267a2981ab8b"} Nov 01 00:40:17.390316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.390270 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-metrics-certs\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.390469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.390455 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a58ae1-5f3a-4556-a74d-5c3420fc172b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.390521 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.390476 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.390521 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.390493 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpch9\" (UniqueName: \"kubernetes.io/projected/87a58ae1-5f3a-4556-a74d-5c3420fc172b-kube-api-access-hpch9\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.411986 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.411897 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-30fb19-serving-cert\"" Nov 01 00:40:17.414364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.414317 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7"] Nov 01 00:40:17.491401 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.491367 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a58ae1-5f3a-4556-a74d-5c3420fc172b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.491569 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.491407 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.491569 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.491433 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpch9\" (UniqueName: \"kubernetes.io/projected/87a58ae1-5f3a-4556-a74d-5c3420fc172b-kube-api-access-hpch9\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.491569 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.491523 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-metrics-certs\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.491747 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:17.491599 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-serving-cert: secret "maas-api-e2e-worker-w1-30fb19-serving-cert" not found Nov 01 00:40:17.491747 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:17.491709 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls podName:87a58ae1-5f3a-4556-a74d-5c3420fc172b nodeName:}" failed. No retries permitted until 2026-11-01 00:40:17.991659582 +0000 UTC m=+962.866880591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls") pod "maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" (UID: "87a58ae1-5f3a-4556-a74d-5c3420fc172b") : secret "maas-api-e2e-worker-w1-30fb19-serving-cert" not found Nov 01 00:40:17.492132 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.492107 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a58ae1-5f3a-4556-a74d-5c3420fc172b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.494991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.494962 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-metrics-certs\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.553452 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.553413 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpch9\" (UniqueName: \"kubernetes.io/projected/87a58ae1-5f3a-4556-a74d-5c3420fc172b-kube-api-access-hpch9\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.616201 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.616116 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:17.818242 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.818194 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:17.997364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.997244 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:17.999962 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:17.999932 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:18.270617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:18.270533 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:18.646128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:18.646095 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:18.658120 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:18.658097 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7"] Nov 01 00:40:18.662302 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:18.662251 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a58ae1_5f3a_4556_a74d_5c3420fc172b.slice/crio-2a71e3ef22975daf6d71497203f83e4ea23947b4fa8f3f7050237b9c21f07710 WatchSource:0}: Error finding container 2a71e3ef22975daf6d71497203f83e4ea23947b4fa8f3f7050237b9c21f07710: Status 404 returned error can't find the container with id 2a71e3ef22975daf6d71497203f83e4ea23947b4fa8f3f7050237b9c21f07710 Nov 01 00:40:18.879644 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:18.879613 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:19.400880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:19.400838 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" event={"ID":"87a58ae1-5f3a-4556-a74d-5c3420fc172b","Type":"ContainerStarted","Data":"2a71e3ef22975daf6d71497203f83e4ea23947b4fa8f3f7050237b9c21f07710"} Nov 01 00:40:19.507473 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:19.507436 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:40:20.407828 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.407790 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" event={"ID":"87a58ae1-5f3a-4556-a74d-5c3420fc172b","Type":"ContainerStarted","Data":"936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff"} Nov 01 00:40:20.408336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.407893 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:20.409398 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.409371 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" event={"ID":"751ff7e8-5d91-4386-8882-c7d75d4e668f","Type":"ContainerStarted","Data":"74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5"} Nov 01 00:40:20.409539 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.409470 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:20.410933 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.410913 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" event={"ID":"675d3e25-4ab9-46be-878a-d049d7e50e58","Type":"ContainerStarted","Data":"6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c"} Nov 01 00:40:20.411062 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.411048 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:20.542842 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.542790 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" podStartSLOduration=2.6676226610000002 podStartE2EDuration="3.542775178s" totalImagesPullingTime="875.152517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:20.541458325 +0000 UTC m=+965.416679339" watchObservedRunningTime="2026-11-01 00:40:20.542775178 +0000 UTC m=+965.417996193" Nov 01 00:40:20.662438 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.662269 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" podStartSLOduration=1.761334968 podStartE2EDuration="4.662248476s" totalImagesPullingTime="2.900913508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:20.658690906 +0000 UTC m=+965.533911921" watchObservedRunningTime="2026-11-01 00:40:20.662248476 +0000 UTC m=+965.537469492" Nov 01 00:40:20.777837 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:20.777755 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" podStartSLOduration=1.814558263 podStartE2EDuration="4.777733742s" totalImagesPullingTime="2.963175479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:20.776429552 +0000 UTC m=+965.651650566" watchObservedRunningTime="2026-11-01 00:40:20.777733742 +0000 UTC m=+965.652954758" Nov 01 00:40:21.122526 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:21.122493 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:21.344779 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:21.344743 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:22.725666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.725631 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk"] Nov 01 00:40:22.729595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.729573 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.782787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.782747 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-43a0cf-serving-cert\"" Nov 01 00:40:22.836682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.836649 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk"] Nov 01 00:40:22.875256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.874849 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e957015c-79f2-4e43-b7dd-cc89e32169ba-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.875256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.874935 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.875256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.874995 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-metrics-certs\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.875256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.875126 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6knh\" (UniqueName: \"kubernetes.io/projected/e957015c-79f2-4e43-b7dd-cc89e32169ba-kube-api-access-q6knh\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.981469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.977933 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e957015c-79f2-4e43-b7dd-cc89e32169ba-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.981469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.978817 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.981469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.978898 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-metrics-certs\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.981469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.979029 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6knh\" (UniqueName: \"kubernetes.io/projected/e957015c-79f2-4e43-b7dd-cc89e32169ba-kube-api-access-q6knh\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.981469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.980403 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e957015c-79f2-4e43-b7dd-cc89e32169ba-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.981469 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:22.980522 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-serving-cert: secret "maas-api-e2e-worker-w4-43a0cf-serving-cert" not found Nov 01 00:40:22.981469 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:22.980593 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls podName:e957015c-79f2-4e43-b7dd-cc89e32169ba nodeName:}" failed. No retries permitted until 2026-11-01 00:40:23.480573017 +0000 UTC m=+968.355794014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls") pod "maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" (UID: "e957015c-79f2-4e43-b7dd-cc89e32169ba") : secret "maas-api-e2e-worker-w4-43a0cf-serving-cert" not found Nov 01 00:40:22.984245 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.984129 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-metrics-certs\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:22.990255 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:22.990231 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:23.043349 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:23.043319 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6knh\" (UniqueName: \"kubernetes.io/projected/e957015c-79f2-4e43-b7dd-cc89e32169ba-kube-api-access-q6knh\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:23.199275 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:23.199224 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:23.408674 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:23.408601 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:23.484916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:23.484882 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:23.487486 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:23.487458 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls\") pod \"maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:23.643096 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:23.643057 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:24.043494 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:24.043470 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk"] Nov 01 00:40:24.045258 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:24.045231 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode957015c_79f2_4e43_b7dd_cc89e32169ba.slice/crio-025320ff6e61e1578fca14d8a25823259c46a647d739caa7a53360362b5cee93 WatchSource:0}: Error finding container 025320ff6e61e1578fca14d8a25823259c46a647d739caa7a53360362b5cee93: Status 404 returned error can't find the container with id 025320ff6e61e1578fca14d8a25823259c46a647d739caa7a53360362b5cee93 Nov 01 00:40:24.432779 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:24.432697 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" event={"ID":"e957015c-79f2-4e43-b7dd-cc89e32169ba","Type":"ContainerStarted","Data":"025320ff6e61e1578fca14d8a25823259c46a647d739caa7a53360362b5cee93"} Nov 01 00:40:25.439849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:25.439816 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" event={"ID":"e957015c-79f2-4e43-b7dd-cc89e32169ba","Type":"ContainerStarted","Data":"60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b"} Nov 01 00:40:25.440269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:25.439890 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:25.563322 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:25.561709 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" podStartSLOduration=2.723560854 podStartE2EDuration="3.561687455s" totalImagesPullingTime="838.126601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:25.558824531 +0000 UTC m=+970.434045552" watchObservedRunningTime="2026-11-01 00:40:25.561687455 +0000 UTC m=+970.436908471" Nov 01 00:40:25.606953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:25.606919 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:26.178011 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:26.177916 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:26.391593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:26.391564 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:26.423129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:26.423104 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 00:40:28.285960 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:28.285927 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:28.515033 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:28.514992 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:28.743259 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:28.743181 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:28.950831 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:28.950797 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:30.475794 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:30.475762 6036 generic.go:356] "Generic (PLEG): container finished" podID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerID="f3c250fde8915e6f0cee6e25a76ccab2a463ea18acbf5a23915e9a4e873772fd" exitCode=0 Nov 01 00:40:30.476222 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:30.475840 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" event={"ID":"3e5da06e-9f90-4586-b65e-fa6173da4969","Type":"ContainerDied","Data":"f3c250fde8915e6f0cee6e25a76ccab2a463ea18acbf5a23915e9a4e873772fd"} Nov 01 00:40:30.476266 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:30.476229 6036 scope.go:122] "RemoveContainer" containerID="f3c250fde8915e6f0cee6e25a76ccab2a463ea18acbf5a23915e9a4e873772fd" Nov 01 00:40:30.477617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:30.477592 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:30.477676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:30.477656 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:30.477711 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:30.477688 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:31.419754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:31.419721 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 00:40:31.419962 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:31.419895 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 00:40:31.500141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:31.500105 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 00:40:31.502454 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:31.502426 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" event={"ID":"3e5da06e-9f90-4586-b65e-fa6173da4969","Type":"ContainerStarted","Data":"b2c7cdd6d72c6feba80759c81d29622e28d7cf81180aa399f5c3c8d037a35317"} Nov 01 00:40:31.618723 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:31.617846 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:31.618916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:31.618785 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 00:40:31.618916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:31.618854 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 00:40:32.618888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:32.618840 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 00:40:32.619308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:32.618904 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 00:40:33.423349 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.423306 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp"] Nov 01 00:40:33.430101 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.430074 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.497461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.497428 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-5a9c19-serving-cert\"" Nov 01 00:40:33.528892 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.528854 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-metrics-certs\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.529051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.528902 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a961708-53e1-444c-ac49-db165a3acf56-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.529051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.528951 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n9d\" (UniqueName: \"kubernetes.io/projected/7a961708-53e1-444c-ac49-db165a3acf56-kube-api-access-67n9d\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.529051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.528991 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.571433 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.561484 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp"] Nov 01 00:40:33.618619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.618579 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 00:40:33.618818 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.618626 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 00:40:33.630201 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.630171 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-metrics-certs\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.630573 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.630211 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a961708-53e1-444c-ac49-db165a3acf56-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.630573 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.630279 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n9d\" (UniqueName: \"kubernetes.io/projected/7a961708-53e1-444c-ac49-db165a3acf56-kube-api-access-67n9d\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.630573 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.630367 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.630573 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:33.630517 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-serving-cert: secret "maas-api-e2e-worker-w3-5a9c19-serving-cert" not found Nov 01 00:40:33.630706 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:33.630581 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls podName:7a961708-53e1-444c-ac49-db165a3acf56 nodeName:}" failed. No retries permitted until 2026-11-01 00:40:34.13055816 +0000 UTC m=+979.005779174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls") pod "maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" (UID: "7a961708-53e1-444c-ac49-db165a3acf56") : secret "maas-api-e2e-worker-w3-5a9c19-serving-cert" not found Nov 01 00:40:33.631201 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.631180 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a961708-53e1-444c-ac49-db165a3acf56-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.632798 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.632773 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-metrics-certs\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.688366 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.686205 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:33.701851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.701677 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n9d\" (UniqueName: \"kubernetes.io/projected/7a961708-53e1-444c-ac49-db165a3acf56-kube-api-access-67n9d\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:33.929757 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:33.929711 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:34.136158 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:34.136116 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:34.138820 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:34.138796 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:34.165313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:34.160750 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:34.317408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:34.317375 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:34.343941 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:34.343910 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:34.618607 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:34.618574 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 00:40:34.618772 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:34.618638 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 00:40:34.787414 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:34.787376 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp"] Nov 01 00:40:34.801684 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:34.801655 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a961708_53e1_444c_ac49_db165a3acf56.slice/crio-dd0b35161ff1d183257abb6c01899ca266ddd63af8f3da5fb2199e2b0d33abb0 WatchSource:0}: Error finding container dd0b35161ff1d183257abb6c01899ca266ddd63af8f3da5fb2199e2b0d33abb0: Status 404 returned error can't find the container with id dd0b35161ff1d183257abb6c01899ca266ddd63af8f3da5fb2199e2b0d33abb0 Nov 01 00:40:35.525557 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:35.525526 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" event={"ID":"7a961708-53e1-444c-ac49-db165a3acf56","Type":"ContainerStarted","Data":"dd0b35161ff1d183257abb6c01899ca266ddd63af8f3da5fb2199e2b0d33abb0"} Nov 01 00:40:35.618428 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:35.618390 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 00:40:35.618604 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:35.618446 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 00:40:36.097969 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:36.097855 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:36.341589 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:36.341557 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:36.533758 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:36.533721 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" event={"ID":"7a961708-53e1-444c-ac49-db165a3acf56","Type":"ContainerStarted","Data":"8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27"} Nov 01 00:40:36.533995 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:36.533964 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:36.618062 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:36.618031 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 00:40:36.618222 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:36.618088 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 00:40:36.656894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:36.656831 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" podStartSLOduration=2.6543877350000002 podStartE2EDuration="3.656809691s" totalImagesPullingTime="1.002421956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:36.655261232 +0000 UTC m=+981.530482273" watchObservedRunningTime="2026-11-01 00:40:36.656809691 +0000 UTC m=+981.532030708" Nov 01 00:40:37.623757 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:37.623729 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:37.625007 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:37.624978 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:40:39.137975 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:39.137944 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:39.552449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:39.552415 6036 generic.go:356] "Generic (PLEG): container finished" podID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerID="4c92376e8ea155a66c9de70f76d169079d6f8d5d4605fa260f76a9d69789dcab" exitCode=0 Nov 01 00:40:39.552657 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:39.552493 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" event={"ID":"75c9ed19-e38a-4bf4-859b-9dde6bf7901e","Type":"ContainerDied","Data":"4c92376e8ea155a66c9de70f76d169079d6f8d5d4605fa260f76a9d69789dcab"} Nov 01 00:40:39.552900 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:39.552885 6036 scope.go:122] "RemoveContainer" containerID="4c92376e8ea155a66c9de70f76d169079d6f8d5d4605fa260f76a9d69789dcab" Nov 01 00:40:39.554367 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:39.554341 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:39.554440 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:39.554414 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:39.554476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:39.554446 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:40.601793 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:40.601755 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" event={"ID":"75c9ed19-e38a-4bf4-859b-9dde6bf7901e","Type":"ContainerStarted","Data":"d51a918e5c2a7c00fb7de93e1567ff916919182b54108078b63c08f843030ce3"} Nov 01 00:40:41.208651 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:41.208607 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:41.208849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:41.208825 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 00:40:41.208900 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:41.208867 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 00:40:42.209084 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:42.209046 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 00:40:42.209519 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:42.209099 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 00:40:42.565311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:42.560455 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 00:40:43.208912 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:43.208873 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 00:40:43.209134 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:43.208956 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 00:40:43.921707 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:43.921676 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc"] Nov 01 00:40:43.925931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:43.925909 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:43.983300 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:43.983215 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-32439f-serving-cert\"" Nov 01 00:40:44.024245 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.024204 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.024245 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.024248 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.024511 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.024389 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whgcr\" (UniqueName: \"kubernetes.io/projected/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-kube-api-access-whgcr\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.024511 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.024437 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-metrics-certs\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.034985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.034955 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc"] Nov 01 00:40:44.125424 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.125393 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.125565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.125446 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.125565 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:44.125516 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-serving-cert: secret "maas-api-e2e-worker-w0-32439f-serving-cert" not found Nov 01 00:40:44.125565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.125525 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whgcr\" (UniqueName: \"kubernetes.io/projected/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-kube-api-access-whgcr\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.125737 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:44.125596 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls podName:7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4 nodeName:}" failed. No retries permitted until 2026-11-01 00:40:44.625571705 +0000 UTC m=+989.500792721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls") pod "maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" (UID: "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4") : secret "maas-api-e2e-worker-w0-32439f-serving-cert" not found Nov 01 00:40:44.125737 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.125645 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-metrics-certs\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.126068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.126045 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.129743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.129716 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-metrics-certs\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.202801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.202665 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:44.204602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.204574 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whgcr\" (UniqueName: \"kubernetes.io/projected/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-kube-api-access-whgcr\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.208568 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.208545 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 00:40:44.208674 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.208596 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 00:40:44.413681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.413647 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:44.629431 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.629403 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.632882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.632861 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls\") pod \"maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:44.839276 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:44.839238 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:45.208964 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:45.208875 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 00:40:45.208964 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:45.208939 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 00:40:45.296312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:45.295693 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc"] Nov 01 00:40:45.302112 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:40:45.302084 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1c4e42_1f20_42b8_8144_6bb8f3c0c1c4.slice/crio-e49a3fde4f9c6183b90db13cd02577f1b1434b869fc933254e53236cea3bad3b WatchSource:0}: Error finding container e49a3fde4f9c6183b90db13cd02577f1b1434b869fc933254e53236cea3bad3b: Status 404 returned error can't find the container with id e49a3fde4f9c6183b90db13cd02577f1b1434b869fc933254e53236cea3bad3b Nov 01 00:40:45.630964 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:45.630928 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" event={"ID":"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4","Type":"ContainerStarted","Data":"e49a3fde4f9c6183b90db13cd02577f1b1434b869fc933254e53236cea3bad3b"} Nov 01 00:40:46.213257 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:46.213228 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:46.213607 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:46.213581 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:46.214372 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:46.214354 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 00:40:46.638337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:46.638301 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" event={"ID":"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4","Type":"ContainerStarted","Data":"61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027"} Nov 01 00:40:46.638822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:46.638360 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:46.759855 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:46.759779 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" podStartSLOduration=2.930941178 podStartE2EDuration="3.759756447s" totalImagesPullingTime="828.815269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:46.758378839 +0000 UTC m=+991.633599853" watchObservedRunningTime="2026-11-01 00:40:46.759756447 +0000 UTC m=+991.634977470" Nov 01 00:40:49.092914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:49.092871 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:49.327731 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:49.327700 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:49.504187 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:40:49.504149 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:40:49.661374 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:49.661340 6036 generic.go:356] "Generic (PLEG): container finished" podID="26a0e24d-d19e-4750-9423-b6f61f034983" containerID="17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c" exitCode=0 Nov 01 00:40:49.661540 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:49.661387 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" event={"ID":"26a0e24d-d19e-4750-9423-b6f61f034983","Type":"ContainerDied","Data":"17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c"} Nov 01 00:40:49.661783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:49.661769 6036 scope.go:122] "RemoveContainer" containerID="17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c" Nov 01 00:40:49.663132 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:49.663105 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:49.663203 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:49.663172 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:49.663238 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:49.663202 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:40:50.321929 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:50.321894 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:50.681061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:50.680979 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" event={"ID":"26a0e24d-d19e-4750-9423-b6f61f034983","Type":"ContainerStarted","Data":"f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38"} Nov 01 00:40:51.565671 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:51.565639 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:51.570429 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:51.570397 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 00:40:51.570670 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:51.570645 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 00:40:52.565729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:52.565690 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 00:40:52.566392 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:52.566359 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 00:40:52.672518 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:52.672491 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 00:40:53.568312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:53.567683 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 00:40:53.568312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:53.567758 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 00:40:54.173050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:54.173015 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:54.569821 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:54.569782 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 00:40:54.570429 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:54.570386 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 00:40:55.564666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:55.564633 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 00:40:55.564882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:55.564688 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 00:40:56.569539 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:56.569506 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:56.570027 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:56.569909 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:56.570695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:56.570673 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 00:40:59.055222 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:59.055189 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:40:59.567651 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:40:59.567615 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:01.512471 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:01.512440 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:02.112102 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:02.112055 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:02.342315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:02.342250 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:04.305780 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:04.305751 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:06.171319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:06.171264 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:06.706136 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:06.706024 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:07.409406 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:07.409365 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:07.640786 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:07.640748 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:09.812348 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:09.812275 6036 generic.go:356] "Generic (PLEG): container finished" podID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerID="73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091" exitCode=0 Nov 01 00:41:09.812348 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:09.812325 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" event={"ID":"5e4a2c30-4b8f-4d6e-90f8-8569f072d908","Type":"ContainerDied","Data":"73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091"} Nov 01 00:41:09.812927 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:09.812761 6036 scope.go:122] "RemoveContainer" containerID="73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091" Nov 01 00:41:09.814048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:09.814021 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:41:09.814118 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:09.814084 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:41:09.814118 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:09.814114 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:41:09.851937 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:09.851850 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:10.819998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:10.819965 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" event={"ID":"5e4a2c30-4b8f-4d6e-90f8-8569f072d908","Type":"ContainerStarted","Data":"a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875"} Nov 01 00:41:11.587145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:11.587110 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:41:11.587372 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:11.587349 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 00:41:11.587453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:11.587430 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 00:41:12.587850 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:12.587812 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 00:41:12.588262 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:12.587881 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 00:41:13.377499 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:13.377444 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:13.586963 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:13.586929 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 00:41:13.587145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:13.586979 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 00:41:14.187897 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:14.187861 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:14.511510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:14.511411 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:14.587830 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:14.587786 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 00:41:14.588002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:14.587863 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 00:41:15.337865 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:15.337802 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:15.587805 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:15.587766 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 00:41:15.588004 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:15.587823 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 00:41:16.591524 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:16.591494 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:41:16.592007 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:16.591868 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:41:16.592578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:16.592557 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 00:41:17.518475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:17.518437 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:18.087056 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:18.086996 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:18.370733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:18.370646 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:19.503416 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:41:19.503371 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:41:20.696741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:20.696709 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:22.710497 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:22.710464 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:23.415746 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:23.415703 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:23.681190 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:23.681099 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:26.046549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:26.046510 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:26.394660 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:26.394570 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:29.116420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.116386 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2"] Nov 01 00:41:29.122000 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.121968 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.175657 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.175617 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-tmp-dir\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.175837 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.175671 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-model-cache\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.175837 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.175784 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-home\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.175837 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.175817 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbq8\" (UniqueName: \"kubernetes.io/projected/f75a426a-af3c-4361-b7ba-a3f06345820f-kube-api-access-8zbq8\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.175963 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.175865 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-dshm\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.175997 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.175973 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75a426a-af3c-4361-b7ba-a3f06345820f-tls-certs\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.182544 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.182513 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"openshift-service-ca.crt\"" Nov 01 00:41:29.183866 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.183849 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"kube-root-ca.crt\"" Nov 01 00:41:29.193856 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.193827 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"default-dockercfg-b2lgb\"" Nov 01 00:41:29.233362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.233329 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"simulator-w1-30fb19-kserve-self-signed-certs\"" Nov 01 00:41:29.236450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.236430 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2"] Nov 01 00:41:29.277359 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.277314 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75a426a-af3c-4361-b7ba-a3f06345820f-tls-certs\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.277580 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.277549 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-tmp-dir\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.277659 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.277584 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-model-cache\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.277659 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.277621 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-home\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.277769 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.277655 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbq8\" (UniqueName: \"kubernetes.io/projected/f75a426a-af3c-4361-b7ba-a3f06345820f-kube-api-access-8zbq8\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.277769 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.277702 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-dshm\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.278336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.278164 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-model-cache\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.278662 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.278641 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-home\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.278905 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.278786 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-tmp-dir\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.280430 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.280411 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-dshm\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.280716 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.280694 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75a426a-af3c-4361-b7ba-a3f06345820f-tls-certs\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.312485 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.312443 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:29.345737 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.345705 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbq8\" (UniqueName: \"kubernetes.io/projected/f75a426a-af3c-4361-b7ba-a3f06345820f-kube-api-access-8zbq8\") pod \"simulator-w1-30fb19-kserve-b4c576679-d7sc2\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.441378 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.441253 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:29.830038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.830009 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2"] Nov 01 00:41:29.830795 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:41:29.830768 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf75a426a_af3c_4361_b7ba_a3f06345820f.slice/crio-dfbc49d6496d1bfe863b43d4daf64397560f2a114914350aefd60ccd9e422cae WatchSource:0}: Error finding container dfbc49d6496d1bfe863b43d4daf64397560f2a114914350aefd60ccd9e422cae: Status 404 returned error can't find the container with id dfbc49d6496d1bfe863b43d4daf64397560f2a114914350aefd60ccd9e422cae Nov 01 00:41:29.944783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.944750 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" event={"ID":"f75a426a-af3c-4361-b7ba-a3f06345820f","Type":"ContainerStarted","Data":"5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e"} Nov 01 00:41:29.944783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:29.944790 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" event={"ID":"f75a426a-af3c-4361-b7ba-a3f06345820f","Type":"ContainerStarted","Data":"dfbc49d6496d1bfe863b43d4daf64397560f2a114914350aefd60ccd9e422cae"} Nov 01 00:41:30.065631 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:30.065583 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podStartSLOduration=1.065569203 podStartE2EDuration="1.065569203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:41:30.064339281 +0000 UTC m=+1034.939560293" watchObservedRunningTime="2026-11-01 00:41:30.065569203 +0000 UTC m=+1034.940790218" Nov 01 00:41:30.441848 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:30.441817 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:31.335470 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:31.335440 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:33.224569 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:33.224533 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:33.825529 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:33.825493 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:34.086003 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:34.085918 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:36.829698 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:36.829668 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:37.427461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:37.427425 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:38.031453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:38.031425 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:38.370820 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:38.370788 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:39.441706 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:39.441666 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:39.455358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:39.455330 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:40.015164 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:40.015134 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:41:42.243599 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:42.243566 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:45.961151 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:45.961121 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:47.160655 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:47.160618 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:49.503981 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:41:49.503941 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:41:51.201300 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:51.201253 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:52.151893 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:52.151859 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:41:57.235267 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:41:57.235231 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:02.141155 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:02.141121 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:05.965035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:05.964998 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8"] Nov 01 00:42:05.968834 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:05.968816 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.072659 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.072619 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"openshift-service-ca.crt\"" Nov 01 00:42:06.072832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.072670 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"simulator-w0-32439f-kserve-self-signed-certs\"" Nov 01 00:42:06.072832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.072762 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"kube-root-ca.crt\"" Nov 01 00:42:06.072832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.072827 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"default-dockercfg-d59kz\"" Nov 01 00:42:06.074992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.074973 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8"] Nov 01 00:42:06.101202 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.101165 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-home\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.101427 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.101217 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-tmp-dir\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.101502 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.101439 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477b9668-6573-401c-b659-5feeec968d5a-tls-certs\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.101578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.101554 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bh9b\" (UniqueName: \"kubernetes.io/projected/477b9668-6573-401c-b659-5feeec968d5a-kube-api-access-7bh9b\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.101686 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.101666 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-dshm\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.101750 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.101722 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-model-cache\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.124239 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.124209 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:06.202774 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.202738 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-model-cache\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.202968 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.202802 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-home\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.202968 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.202825 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-tmp-dir\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.202968 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.202876 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477b9668-6573-401c-b659-5feeec968d5a-tls-certs\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.202968 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.202933 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bh9b\" (UniqueName: \"kubernetes.io/projected/477b9668-6573-401c-b659-5feeec968d5a-kube-api-access-7bh9b\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.203207 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.202980 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-dshm\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.203265 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.203199 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-model-cache\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.203354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.203271 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-tmp-dir\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.203354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.203276 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-home\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.205482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.205460 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-dshm\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.205678 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.205662 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477b9668-6573-401c-b659-5feeec968d5a-tls-certs\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.265255 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.265182 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bh9b\" (UniqueName: \"kubernetes.io/projected/477b9668-6573-401c-b659-5feeec968d5a-kube-api-access-7bh9b\") pod \"simulator-w0-32439f-kserve-77dfd67cdb-h44p8\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.282150 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.282120 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:06.661377 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:06.661352 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8"] Nov 01 00:42:06.663710 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:42:06.663685 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod477b9668_6573_401c_b659_5feeec968d5a.slice/crio-f25aea948e62615cd42f27c57ae0509b8013b429b9f2c2d65adffd1a867bfdf6 WatchSource:0}: Error finding container f25aea948e62615cd42f27c57ae0509b8013b429b9f2c2d65adffd1a867bfdf6: Status 404 returned error can't find the container with id f25aea948e62615cd42f27c57ae0509b8013b429b9f2c2d65adffd1a867bfdf6 Nov 01 00:42:07.169375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:07.169335 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" event={"ID":"477b9668-6573-401c-b659-5feeec968d5a","Type":"ContainerStarted","Data":"4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71"} Nov 01 00:42:07.169375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:07.169382 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" event={"ID":"477b9668-6573-401c-b659-5feeec968d5a","Type":"ContainerStarted","Data":"f25aea948e62615cd42f27c57ae0509b8013b429b9f2c2d65adffd1a867bfdf6"} Nov 01 00:42:07.177421 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:07.177391 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:07.282680 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:07.282651 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:07.289466 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:07.289419 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podStartSLOduration=2.289405698 podStartE2EDuration="2.289405698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:42:07.287853495 +0000 UTC m=+1072.163074511" watchObservedRunningTime="2026-11-01 00:42:07.289405698 +0000 UTC m=+1072.164626843" Nov 01 00:42:12.144576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:12.144540 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:16.282816 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:16.282777 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:16.295747 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:16.295719 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:16.760809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:16.760721 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:17.166062 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:17.166030 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:17.232109 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:17.232082 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:42:19.504162 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:42:19.504127 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:42:19.708120 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.708089 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq"] Nov 01 00:42:19.712147 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.712129 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.753410 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.753378 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnppj\" (UniqueName: \"kubernetes.io/projected/af64c3fe-059e-4f14-9eb0-910b05dae0b5-kube-api-access-gnppj\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.753578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.753417 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tls-certs\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.753578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.753525 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tmp-dir\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.753680 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.753579 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-dshm\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.753680 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.753606 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-model-cache\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.753804 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.753783 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-home\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.818888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.818846 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"kube-root-ca.crt\"" Nov 01 00:42:19.819106 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.818913 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"openshift-service-ca.crt\"" Nov 01 00:42:19.819106 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.818986 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"premium-w6-22d7b2-kserve-self-signed-certs\"" Nov 01 00:42:19.819243 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.819202 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"default-dockercfg-xn6ld\"" Nov 01 00:42:19.820957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.820939 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq"] Nov 01 00:42:19.855207 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.855116 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnppj\" (UniqueName: \"kubernetes.io/projected/af64c3fe-059e-4f14-9eb0-910b05dae0b5-kube-api-access-gnppj\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.855207 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.855148 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tls-certs\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.855207 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.855197 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tmp-dir\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.855578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.855240 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-dshm\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.855578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.855268 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-model-cache\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.855578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.855339 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-home\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.855749 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.855654 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tmp-dir\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.856177 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.856134 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-model-cache\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.859962 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.858274 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-home\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.859962 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.858575 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tls-certs\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.863431 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.863402 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-dshm\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:19.873876 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.873828 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:19.927048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:19.927005 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnppj\" (UniqueName: \"kubernetes.io/projected/af64c3fe-059e-4f14-9eb0-910b05dae0b5-kube-api-access-gnppj\") pod \"premium-w6-22d7b2-kserve-7bcd85cf79-mnctq\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:20.024418 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:20.024382 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:20.400011 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:20.399984 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq"] Nov 01 00:42:20.402155 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:42:20.402128 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf64c3fe_059e_4f14_9eb0_910b05dae0b5.slice/crio-8740b22fb0abd29231027250c3eb0323acc9cae8aae6c02d9ac946ea2da0667e WatchSource:0}: Error finding container 8740b22fb0abd29231027250c3eb0323acc9cae8aae6c02d9ac946ea2da0667e: Status 404 returned error can't find the container with id 8740b22fb0abd29231027250c3eb0323acc9cae8aae6c02d9ac946ea2da0667e Nov 01 00:42:21.257841 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:21.257803 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" event={"ID":"af64c3fe-059e-4f14-9eb0-910b05dae0b5","Type":"ContainerStarted","Data":"67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584"} Nov 01 00:42:21.257841 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:21.257833 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" event={"ID":"af64c3fe-059e-4f14-9eb0-910b05dae0b5","Type":"ContainerStarted","Data":"8740b22fb0abd29231027250c3eb0323acc9cae8aae6c02d9ac946ea2da0667e"} Nov 01 00:42:21.377151 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:21.377099 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podStartSLOduration=2.377086664 podStartE2EDuration="2.377086664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:42:21.376377002 +0000 UTC m=+1086.251598016" watchObservedRunningTime="2026-11-01 00:42:21.377086664 +0000 UTC m=+1086.252307679" Nov 01 00:42:22.024677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:22.024593 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:22.083315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:22.083259 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:26.380927 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:26.380839 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:26.936578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:26.936540 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:30.025250 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:30.025216 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:30.037964 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:30.037936 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:30.325475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:30.325444 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:42:31.840528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:31.840493 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:35.261115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:35.261081 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:36.640351 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:36.640314 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:38.764860 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:38.764817 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:41.519858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:41.519827 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:46.432822 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:46.432790 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:48.940418 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:48.940380 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:49.505125 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:42:49.505088 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:42:51.248335 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:51.248280 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:56.128025 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:56.127994 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:42:57.478731 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:42:57.478695 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:00.946759 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:00.946720 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:05.760666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:05.760633 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:10.565970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:10.565930 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:13.152666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.152630 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv"] Nov 01 00:43:13.156718 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.156697 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.212487 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.212453 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"openshift-service-ca.crt\"" Nov 01 00:43:13.247905 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.247876 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-home\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.248049 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.247977 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-tmp-dir\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.248049 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.248025 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-model-cache\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.248150 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.248075 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-dshm\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.248257 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.248231 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0460f-fcb9-48c2-a465-a9a0845177e6-tls-certs\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.248346 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.248329 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrtr\" (UniqueName: \"kubernetes.io/projected/bec0460f-fcb9-48c2-a465-a9a0845177e6-kube-api-access-knrtr\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.262940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.262916 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"kube-root-ca.crt\"" Nov 01 00:43:13.263177 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.263158 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"default-dockercfg-g7cxq\"" Nov 01 00:43:13.263237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.263178 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4143b7\"/\"premium-w2-4143b7-kserve-self-signed-certs\"" Nov 01 00:43:13.265147 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.265131 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv"] Nov 01 00:43:13.314531 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.314486 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:13.349355 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349259 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrtr\" (UniqueName: \"kubernetes.io/projected/bec0460f-fcb9-48c2-a465-a9a0845177e6-kube-api-access-knrtr\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.349560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349364 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-home\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.349560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349414 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-tmp-dir\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.349560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349457 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-model-cache\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.349560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349502 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-dshm\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.349774 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349635 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0460f-fcb9-48c2-a465-a9a0845177e6-tls-certs\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.349832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349798 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-home\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.349893 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349830 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-tmp-dir\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.349893 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.349869 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-model-cache\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.351835 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.351811 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-dshm\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.352248 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.352228 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0460f-fcb9-48c2-a465-a9a0845177e6-tls-certs\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.415939 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.415861 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrtr\" (UniqueName: \"kubernetes.io/projected/bec0460f-fcb9-48c2-a465-a9a0845177e6-kube-api-access-knrtr\") pod \"premium-w2-4143b7-kserve-f4c88f8f9-mnsrv\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.469038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.469003 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:13.850717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:13.850614 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv"] Nov 01 00:43:13.853276 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:43:13.853242 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbec0460f_fcb9_48c2_a465_a9a0845177e6.slice/crio-bddb25c40b2319930d240ab3d59e987b32ce400484af7c0b5c7c49d38f0fada0 WatchSource:0}: Error finding container bddb25c40b2319930d240ab3d59e987b32ce400484af7c0b5c7c49d38f0fada0: Status 404 returned error can't find the container with id bddb25c40b2319930d240ab3d59e987b32ce400484af7c0b5c7c49d38f0fada0 Nov 01 00:43:14.564348 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:14.564312 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" event={"ID":"bec0460f-fcb9-48c2-a465-a9a0845177e6","Type":"ContainerStarted","Data":"d781f0ed6c8841f30ebfe496ed7966e720c91d7030b04e04e859160f1a1351a5"} Nov 01 00:43:14.564754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:14.564359 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" event={"ID":"bec0460f-fcb9-48c2-a465-a9a0845177e6","Type":"ContainerStarted","Data":"bddb25c40b2319930d240ab3d59e987b32ce400484af7c0b5c7c49d38f0fada0"} Nov 01 00:43:14.682476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:14.682412 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podStartSLOduration=1.682395923 podStartE2EDuration="1.682395923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:43:14.682026765 +0000 UTC m=+1139.557247780" watchObservedRunningTime="2026-11-01 00:43:14.682395923 +0000 UTC m=+1139.557616945" Nov 01 00:43:15.460772 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:15.460739 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:15.469140 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:15.469120 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:19.503932 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:43:19.503889 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:43:20.262341 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:20.262304 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:23.469966 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:23.469926 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:23.482487 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:23.482463 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:23.625494 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:23.625468 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:43:25.150775 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:25.150737 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:26.660533 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:26.660500 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:28.794944 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:28.794904 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:29.940957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:29.940926 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:34.766752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:34.766714 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:34.980934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:34.980899 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:39.062125 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:39.062085 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:39.321022 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:39.320935 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:39.671080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:39.670980 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:44.566108 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:44.566071 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:47.072755 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.072718 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j"] Nov 01 00:43:47.076833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.076808 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.140969 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.140931 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tmp-dir\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.141158 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.141044 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tls-certs\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.141158 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.141128 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rrdv\" (UniqueName: \"kubernetes.io/projected/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-kube-api-access-6rrdv\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.141305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.141195 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-model-cache\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.141305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.141223 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-dshm\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.141305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.141267 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-home\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.182437 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.182398 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"openshift-service-ca.crt\"" Nov 01 00:43:47.182437 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.182398 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"kube-root-ca.crt\"" Nov 01 00:43:47.182437 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.182406 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"default-dockercfg-xqcln\"" Nov 01 00:43:47.182706 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.182563 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"premium-w5-948e22-kserve-self-signed-certs\"" Nov 01 00:43:47.185142 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.185119 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j"] Nov 01 00:43:47.228358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.228328 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:47.242663 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.242636 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tls-certs\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.242825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.242711 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rrdv\" (UniqueName: \"kubernetes.io/projected/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-kube-api-access-6rrdv\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.242825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.242739 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-model-cache\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.242825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.242759 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-dshm\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.242825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.242810 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-home\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.243010 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.242885 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tmp-dir\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.243382 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.243338 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-home\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.243382 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.243377 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tmp-dir\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.243612 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.243400 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-model-cache\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.245370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.245343 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tls-certs\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.245551 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.245533 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-dshm\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.305364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.305336 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rrdv\" (UniqueName: \"kubernetes.io/projected/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-kube-api-access-6rrdv\") pod \"premium-w5-948e22-kserve-7d57d99467-n9l7j\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.388921 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.388834 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:47.777513 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:47.777482 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j"] Nov 01 00:43:47.779906 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:43:47.779882 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e5fee26_de0f_43bb_b3b2_2a187e9b62a9.slice/crio-fa9ae1c86afb525332ea9f7b2f2f2742a4048cd5551410aa80c081723a85e392 WatchSource:0}: Error finding container fa9ae1c86afb525332ea9f7b2f2f2742a4048cd5551410aa80c081723a85e392: Status 404 returned error can't find the container with id fa9ae1c86afb525332ea9f7b2f2f2742a4048cd5551410aa80c081723a85e392 Nov 01 00:43:48.756377 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:48.756345 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" event={"ID":"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9","Type":"ContainerStarted","Data":"42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea"} Nov 01 00:43:48.756377 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:48.756386 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" event={"ID":"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9","Type":"ContainerStarted","Data":"fa9ae1c86afb525332ea9f7b2f2f2742a4048cd5551410aa80c081723a85e392"} Nov 01 00:43:48.876594 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:48.876539 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podStartSLOduration=1.8765258500000002 podStartE2EDuration="1.87652585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:43:48.874969815 +0000 UTC m=+1173.750190829" watchObservedRunningTime="2026-11-01 00:43:48.87652585 +0000 UTC m=+1173.751746864" Nov 01 00:43:49.389051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:49.389005 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:49.451162 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:49.451132 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:49.504468 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:43:49.504432 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:43:54.242195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:54.242106 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:43:57.389677 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:57.389648 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:57.402583 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:57.402559 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:57.816696 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:57.816670 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:43:59.065172 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:43:59.065142 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:04.034213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:04.034173 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:08.929516 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:08.929485 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:13.740563 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:13.740526 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:18.833433 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:18.833377 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:19.504725 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:44:19.504683 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:44:23.777600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:23.777565 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:27.065907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:27.065869 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:28.839519 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:28.839486 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:33.671108 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:33.671064 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:38.668669 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:38.668624 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:43.567739 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:43.567704 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:48.430247 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:48.430214 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:49.503897 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:44:49.503860 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:44:53.380435 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:53.380390 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:44:58.256463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:44:58.256427 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:00.196801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.196761 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m"] Nov 01 00:45:00.200776 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.200754 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:00.253949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.253921 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:45:00.301692 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.301651 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn79x\" (UniqueName: \"kubernetes.io/projected/2f1c33e9-e54a-494f-89d8-9df44230b0d3-kube-api-access-sn79x\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m\" (UID: \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:00.301902 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.301702 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1c33e9-e54a-494f-89d8-9df44230b0d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m\" (UID: \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:00.402593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.402555 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m"] Nov 01 00:45:00.403240 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.403209 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn79x\" (UniqueName: \"kubernetes.io/projected/2f1c33e9-e54a-494f-89d8-9df44230b0d3-kube-api-access-sn79x\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m\" (UID: \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:00.403419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.403307 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1c33e9-e54a-494f-89d8-9df44230b0d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m\" (UID: \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:00.404272 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.404239 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1c33e9-e54a-494f-89d8-9df44230b0d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m\" (UID: \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:00.465806 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.465732 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn79x\" (UniqueName: \"kubernetes.io/projected/2f1c33e9-e54a-494f-89d8-9df44230b0d3-kube-api-access-sn79x\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m\" (UID: \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:00.514664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.514633 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:00.893782 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.893758 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m"] Nov 01 00:45:00.895974 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:45:00.895948 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1c33e9_e54a_494f_89d8_9df44230b0d3.slice/crio-2534f053e083ff32441fe2f102c2855e14d4b83ebbaa81b014670e6d7a0ab644 WatchSource:0}: Error finding container 2534f053e083ff32441fe2f102c2855e14d4b83ebbaa81b014670e6d7a0ab644: Status 404 returned error can't find the container with id 2534f053e083ff32441fe2f102c2855e14d4b83ebbaa81b014670e6d7a0ab644 Nov 01 00:45:00.897937 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:00.897915 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:45:01.179710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.179617 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" event={"ID":"2f1c33e9-e54a-494f-89d8-9df44230b0d3","Type":"ContainerStarted","Data":"2534f053e083ff32441fe2f102c2855e14d4b83ebbaa81b014670e6d7a0ab644"} Nov 01 00:45:01.785491 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.785454 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk"] Nov 01 00:45:01.789710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.789689 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.842915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.842886 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-30fb19-openshift-default-dockercfg-tm5sb\"" Nov 01 00:45:01.896554 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.896515 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk"] Nov 01 00:45:01.919459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919427 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.919639 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919475 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.919639 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919549 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmtmh\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-kube-api-access-dmtmh\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.919639 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919587 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.919781 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919678 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.919781 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919724 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.919781 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919772 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.919875 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919790 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:01.919910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:01.919873 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.021552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021517 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.021733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021582 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.021733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021632 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.021733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021679 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmtmh\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-kube-api-access-dmtmh\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.021733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021721 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.021941 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021829 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.021941 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021873 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.022039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021942 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.022039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021970 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.022039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.021974 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.022215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.022189 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.022414 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.022388 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.022556 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.022534 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.022616 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.022540 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.024259 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.024232 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.024406 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.024390 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.084002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.083926 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmtmh\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-kube-api-access-dmtmh\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.084377 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.084347 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.103671 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.103628 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:02.541065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.540751 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk"] Nov 01 00:45:02.544374 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:45:02.544341 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d625cd_d8fc_41ea_a34c_9400bbf84dd0.slice/crio-2fbd30e75ec0d95fb3bb927ae006525e130fff78f1a07e93577830dac886b6cb WatchSource:0}: Error finding container 2fbd30e75ec0d95fb3bb927ae006525e130fff78f1a07e93577830dac886b6cb: Status 404 returned error can't find the container with id 2fbd30e75ec0d95fb3bb927ae006525e130fff78f1a07e93577830dac886b6cb Nov 01 00:45:02.547157 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.547119 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:02.547246 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.547205 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:02.547298 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:02.547244 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:03.180196 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:03.180159 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:03.193783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:03.193751 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" event={"ID":"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0","Type":"ContainerStarted","Data":"3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746"} Nov 01 00:45:03.194025 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:03.193804 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" event={"ID":"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0","Type":"ContainerStarted","Data":"2fbd30e75ec0d95fb3bb927ae006525e130fff78f1a07e93577830dac886b6cb"} Nov 01 00:45:03.329787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:03.327698 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" podStartSLOduration=2.327677722 podStartE2EDuration="2.327677722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:03.32692412 +0000 UTC m=+1248.202145136" watchObservedRunningTime="2026-11-01 00:45:03.327677722 +0000 UTC m=+1248.202898737" Nov 01 00:45:04.104515 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:04.104420 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:04.106218 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:04.106183 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 00:45:04.106369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:04.106244 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 00:45:05.105005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:05.104973 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 00:45:05.105396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:05.105038 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 00:45:05.211108 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:05.211074 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" event={"ID":"2f1c33e9-e54a-494f-89d8-9df44230b0d3","Type":"ContainerStarted","Data":"eb5b46384fada5da35c2a0f9d80dc69abc8e3393af5cc31c1b2b204d7f1a2899"} Nov 01 00:45:05.330710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:05.330651 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" podStartSLOduration=1.5107917739999999 podStartE2EDuration="5.330632818s" totalImagesPullingTime="3.819841044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:05.32931311 +0000 UTC m=+1250.204534115" watchObservedRunningTime="2026-11-01 00:45:05.330632818 +0000 UTC m=+1250.205853834" Nov 01 00:45:06.104584 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:06.104505 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 00:45:06.104794 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:06.104771 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 00:45:06.218297 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:06.218252 6036 generic.go:356] "Generic (PLEG): container finished" podID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerID="eb5b46384fada5da35c2a0f9d80dc69abc8e3393af5cc31c1b2b204d7f1a2899" exitCode=60 Nov 01 00:45:06.218710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:06.218342 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" event={"ID":"2f1c33e9-e54a-494f-89d8-9df44230b0d3","Type":"ContainerDied","Data":"eb5b46384fada5da35c2a0f9d80dc69abc8e3393af5cc31c1b2b204d7f1a2899"} Nov 01 00:45:06.218890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:06.218874 6036 scope.go:122] "RemoveContainer" containerID="eb5b46384fada5da35c2a0f9d80dc69abc8e3393af5cc31c1b2b204d7f1a2899" Nov 01 00:45:07.104943 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:07.104905 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 00:45:07.105133 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:07.104962 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 00:45:07.227783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:07.227225 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" event={"ID":"2f1c33e9-e54a-494f-89d8-9df44230b0d3","Type":"ContainerStarted","Data":"919c8df217edde871c78869da6f8c07a20791dd4005d66371e33461ba0a45d28"} Nov 01 00:45:07.978859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:07.978816 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:08.104734 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:08.104696 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 00:45:08.104913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:08.104770 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 00:45:08.236170 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:08.236092 6036 generic.go:356] "Generic (PLEG): container finished" podID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerID="919c8df217edde871c78869da6f8c07a20791dd4005d66371e33461ba0a45d28" exitCode=60 Nov 01 00:45:08.236570 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:08.236200 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" event={"ID":"2f1c33e9-e54a-494f-89d8-9df44230b0d3","Type":"ContainerDied","Data":"919c8df217edde871c78869da6f8c07a20791dd4005d66371e33461ba0a45d28"} Nov 01 00:45:08.236570 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:08.236263 6036 scope.go:122] "RemoveContainer" containerID="eb5b46384fada5da35c2a0f9d80dc69abc8e3393af5cc31c1b2b204d7f1a2899" Nov 01 00:45:08.236663 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:08.236649 6036 scope.go:122] "RemoveContainer" containerID="919c8df217edde871c78869da6f8c07a20791dd4005d66371e33461ba0a45d28" Nov 01 00:45:08.236901 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:45:08.236884 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m_odh-ai-gateway-infra(2f1c33e9-e54a-494f-89d8-9df44230b0d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" Nov 01 00:45:09.245563 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:09.245533 6036 scope.go:122] "RemoveContainer" containerID="919c8df217edde871c78869da6f8c07a20791dd4005d66371e33461ba0a45d28" Nov 01 00:45:09.245989 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:45:09.245828 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m_odh-ai-gateway-infra(2f1c33e9-e54a-494f-89d8-9df44230b0d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" Nov 01 00:45:09.285098 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:09.285067 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:09.286219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:09.286192 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:09.286746 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:09.286728 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 00:45:11.491549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.491519 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g"] Nov 01 00:45:11.496531 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.496512 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.550616 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.550579 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:45:11.563264 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.563234 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xsp\" (UniqueName: \"kubernetes.io/projected/ab75768e-995a-4bc2-877f-3016a9ba66ec-kube-api-access-l5xsp\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.563445 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.563305 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.563530 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.563511 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.603900 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.603867 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g"] Nov 01 00:45:11.665106 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.665071 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xsp\" (UniqueName: \"kubernetes.io/projected/ab75768e-995a-4bc2-877f-3016a9ba66ec-kube-api-access-l5xsp\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.665319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.665114 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.665319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.665186 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.665809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.665787 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.667718 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.667695 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.728080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.728046 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xsp\" (UniqueName: \"kubernetes.io/projected/ab75768e-995a-4bc2-877f-3016a9ba66ec-kube-api-access-l5xsp\") pod \"maas-api-revoke-sub-53cbd772-fqf7g\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:11.808895 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:11.808859 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:12.187514 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:12.187477 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g"] Nov 01 00:45:12.188769 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:45:12.188743 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab75768e_995a_4bc2_877f_3016a9ba66ec.slice/crio-cd316faade20bd07267602297a86129fce67c63591f345d9ae2bd452757d8b06 WatchSource:0}: Error finding container cd316faade20bd07267602297a86129fce67c63591f345d9ae2bd452757d8b06: Status 404 returned error can't find the container with id cd316faade20bd07267602297a86129fce67c63591f345d9ae2bd452757d8b06 Nov 01 00:45:12.265198 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:12.265166 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" event={"ID":"ab75768e-995a-4bc2-877f-3016a9ba66ec","Type":"ContainerStarted","Data":"cd316faade20bd07267602297a86129fce67c63591f345d9ae2bd452757d8b06"} Nov 01 00:45:12.865805 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:12.865768 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:13.272243 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:13.272213 6036 generic.go:356] "Generic (PLEG): container finished" podID="ab75768e-995a-4bc2-877f-3016a9ba66ec" containerID="a9206863426fbf9cb69315228fe1e1afaf5c44fddc69e2a5f44b7980998abe7a" exitCode=0 Nov 01 00:45:13.272438 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:13.272319 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" event={"ID":"ab75768e-995a-4bc2-877f-3016a9ba66ec","Type":"ContainerDied","Data":"a9206863426fbf9cb69315228fe1e1afaf5c44fddc69e2a5f44b7980998abe7a"} Nov 01 00:45:14.431420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.431396 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:14.498638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.498599 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-xks-ca\") pod \"ab75768e-995a-4bc2-877f-3016a9ba66ec\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " Nov 01 00:45:14.498827 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.498660 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab75768e-995a-4bc2-877f-3016a9ba66ec-kube-api-access-l5xsp\" (UniqueName: \"kubernetes.io/projected/ab75768e-995a-4bc2-877f-3016a9ba66ec-kube-api-access-l5xsp\") pod \"ab75768e-995a-4bc2-877f-3016a9ba66ec\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " Nov 01 00:45:14.498827 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.498695 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-openshift-ca\") pod \"ab75768e-995a-4bc2-877f-3016a9ba66ec\" (UID: \"ab75768e-995a-4bc2-877f-3016a9ba66ec\") " Nov 01 00:45:14.499121 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.499097 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-openshift-ca" pod "ab75768e-995a-4bc2-877f-3016a9ba66ec" (UID: "ab75768e-995a-4bc2-877f-3016a9ba66ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:14.501131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.501104 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-xks-ca" pod "ab75768e-995a-4bc2-877f-3016a9ba66ec" (UID: "ab75768e-995a-4bc2-877f-3016a9ba66ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:14.501131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.501125 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab75768e-995a-4bc2-877f-3016a9ba66ec-kube-api-access-l5xsp" pod "ab75768e-995a-4bc2-877f-3016a9ba66ec" (UID: "ab75768e-995a-4bc2-877f-3016a9ba66ec"). InnerVolumeSpecName "kube-api-access-l5xsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:14.600813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.600780 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-openshift-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:14.600813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.600810 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab75768e-995a-4bc2-877f-3016a9ba66ec-maas-api-xks-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:14.600813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:14.600820 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xsp\" (UniqueName: \"kubernetes.io/projected/ab75768e-995a-4bc2-877f-3016a9ba66ec-kube-api-access-l5xsp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:15.284853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:15.284817 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" event={"ID":"ab75768e-995a-4bc2-877f-3016a9ba66ec","Type":"ContainerDied","Data":"cd316faade20bd07267602297a86129fce67c63591f345d9ae2bd452757d8b06"} Nov 01 00:45:15.285019 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:15.284880 6036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd316faade20bd07267602297a86129fce67c63591f345d9ae2bd452757d8b06" Nov 01 00:45:15.285019 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:15.284833 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g" Nov 01 00:45:17.780921 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:17.780882 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:19.504920 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:45:19.504877 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:45:21.243764 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:21.243731 6036 scope.go:122] "RemoveContainer" containerID="919c8df217edde871c78869da6f8c07a20791dd4005d66371e33461ba0a45d28" Nov 01 00:45:22.347775 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:22.347740 6036 generic.go:356] "Generic (PLEG): container finished" podID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerID="3432ec0a9c1726055a34d1dffee9d8138a4c601452155a4bfea0773f4c7b5ebc" exitCode=60 Nov 01 00:45:22.348224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:22.347812 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" event={"ID":"2f1c33e9-e54a-494f-89d8-9df44230b0d3","Type":"ContainerDied","Data":"3432ec0a9c1726055a34d1dffee9d8138a4c601452155a4bfea0773f4c7b5ebc"} Nov 01 00:45:22.348224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:22.347877 6036 scope.go:122] "RemoveContainer" containerID="919c8df217edde871c78869da6f8c07a20791dd4005d66371e33461ba0a45d28" Nov 01 00:45:22.348372 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:22.348256 6036 scope.go:122] "RemoveContainer" containerID="3432ec0a9c1726055a34d1dffee9d8138a4c601452155a4bfea0773f4c7b5ebc" Nov 01 00:45:22.348649 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:45:22.348626 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m_odh-ai-gateway-infra(2f1c33e9-e54a-494f-89d8-9df44230b0d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" Nov 01 00:45:22.828275 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:22.828220 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:23.475876 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:23.475843 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m"] Nov 01 00:45:23.640045 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:23.640014 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:23.712559 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:23.712529 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1c33e9-e54a-494f-89d8-9df44230b0d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1c33e9-e54a-494f-89d8-9df44230b0d3-maas-api-service-ca\") pod \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\" (UID: \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\") " Nov 01 00:45:23.712737 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:23.712570 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1c33e9-e54a-494f-89d8-9df44230b0d3-kube-api-access-sn79x\" (UniqueName: \"kubernetes.io/projected/2f1c33e9-e54a-494f-89d8-9df44230b0d3-kube-api-access-sn79x\") pod \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\" (UID: \"2f1c33e9-e54a-494f-89d8-9df44230b0d3\") " Nov 01 00:45:23.712895 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:23.712860 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1c33e9-e54a-494f-89d8-9df44230b0d3-maas-api-service-ca" pod "2f1c33e9-e54a-494f-89d8-9df44230b0d3" (UID: "2f1c33e9-e54a-494f-89d8-9df44230b0d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:23.713015 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:23.712991 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1c33e9-e54a-494f-89d8-9df44230b0d3-maas-api-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:23.714753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:23.714733 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1c33e9-e54a-494f-89d8-9df44230b0d3-kube-api-access-sn79x" pod "2f1c33e9-e54a-494f-89d8-9df44230b0d3" (UID: "2f1c33e9-e54a-494f-89d8-9df44230b0d3"). InnerVolumeSpecName "kube-api-access-sn79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:23.813832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:23.813798 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn79x\" (UniqueName: \"kubernetes.io/projected/2f1c33e9-e54a-494f-89d8-9df44230b0d3-kube-api-access-sn79x\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:24.354924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:24.354879 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:24.363075 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:24.363044 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" event={"ID":"2f1c33e9-e54a-494f-89d8-9df44230b0d3","Type":"ContainerDied","Data":"2534f053e083ff32441fe2f102c2855e14d4b83ebbaa81b014670e6d7a0ab644"} Nov 01 00:45:24.363241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:24.363104 6036 scope.go:122] "RemoveContainer" containerID="3432ec0a9c1726055a34d1dffee9d8138a4c601452155a4bfea0773f4c7b5ebc" Nov 01 00:45:24.363241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:24.363058 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m" Nov 01 00:45:24.531764 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:24.531730 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m"] Nov 01 00:45:24.582676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:24.582646 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891565-lkj5m"] Nov 01 00:45:25.249834 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:25.249800 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" path="/var/lib/kubelet/pods/2f1c33e9-e54a-494f-89d8-9df44230b0d3/volumes" Nov 01 00:45:25.928559 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:25.928515 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:26.142929 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:26.142880 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:27.359652 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:27.359621 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:27.824318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:27.824223 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:32.751952 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:32.751919 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:37.627940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:37.627909 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:38.935491 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.935462 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc"] Nov 01 00:45:38.935984 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.935970 6036 cpu_manager.go:414] "containerMap: removing container" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:38.936039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.935986 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:38.936039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.936009 6036 cpu_manager.go:414] "containerMap: removing container" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:38.936039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.936014 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:38.936039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.936025 6036 cpu_manager.go:414] "containerMap: removing container" podUID="ab75768e-995a-4bc2-877f-3016a9ba66ec" containerName="revoke-keys" Nov 01 00:45:38.936039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.936031 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab75768e-995a-4bc2-877f-3016a9ba66ec" containerName="revoke-keys" Nov 01 00:45:38.936208 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.936123 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:38.936208 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.936133 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:38.936208 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.936139 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab75768e-995a-4bc2-877f-3016a9ba66ec" containerName="revoke-keys" Nov 01 00:45:38.943443 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:38.943421 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.051828 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.051790 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc"] Nov 01 00:45:39.104393 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104343 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.104593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104410 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jwp\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-kube-api-access-n4jwp\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.104593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104523 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.104593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104557 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.104593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104585 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.104862 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104621 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.104862 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104663 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.104979 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104907 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.104979 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.104961 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.205737 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.205638 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.205737 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.205680 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.205737 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.205701 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206037 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206250 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206229 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206057 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206171 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206128 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206339 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206415 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206486 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206501 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206541 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jwp\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-kube-api-access-n4jwp\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.206819 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.206656 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.208806 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.208780 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.209143 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.209122 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.268757 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.268723 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.269079 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.269059 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jwp\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-kube-api-access-n4jwp\") pod \"maas-default-gateway-openshift-default-67bf986f99-tn2pc\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:39.560925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:39.560228 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:40.163504 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:40.162670 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:40.163504 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:40.162770 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:40.163504 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:40.162814 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:40.164060 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:40.163886 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc"] Nov 01 00:45:40.479193 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:40.479103 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" event={"ID":"0e7b1369-cee3-4dac-bd97-fabb32d2409c","Type":"ContainerStarted","Data":"23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa"} Nov 01 00:45:40.479193 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:40.479164 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" event={"ID":"0e7b1369-cee3-4dac-bd97-fabb32d2409c","Type":"ContainerStarted","Data":"4939a83f905c3c4e09dce6c47cf7b4ff00db6804d257f354caa8fb767a5503c4"} Nov 01 00:45:40.560470 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:40.560419 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:40.603610 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:40.603554 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podStartSLOduration=2.603538984 podStartE2EDuration="2.603538984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:40.6031266 +0000 UTC m=+1285.478347617" watchObservedRunningTime="2026-11-01 00:45:40.603538984 +0000 UTC m=+1285.478759999" Nov 01 00:45:41.570491 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:41.570438 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-tn2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 00:45:41.571021 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:41.570522 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 00:45:42.571845 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:42.571792 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-tn2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 00:45:42.572390 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:42.571874 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 00:45:42.706013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:42.705984 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:43.569309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:43.566706 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-tn2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 00:45:43.569309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:43.566783 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 00:45:44.569313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:44.563781 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-tn2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 00:45:44.569313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:44.563870 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 00:45:45.566311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:45.565713 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-tn2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 00:45:45.566311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:45.565781 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 00:45:46.560913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:46.560876 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-tn2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 00:45:46.561372 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:46.560948 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 00:45:47.561428 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:47.561385 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-tn2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 00:45:47.561856 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:47.561467 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 00:45:47.575271 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:47.575240 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:48.695443 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:48.695408 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:48.696499 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:48.696101 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:48.697028 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:48.697007 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:45:49.506086 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:45:49.506045 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:45:52.441881 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:52.441842 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:54.077239 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.077206 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94"] Nov 01 00:45:54.077803 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.077788 6036 cpu_manager.go:414] "containerMap: removing container" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:54.077852 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.077804 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:54.077915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.077904 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1c33e9-e54a-494f-89d8-9df44230b0d3" containerName="cleanup" Nov 01 00:45:54.081500 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.081481 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.129156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.129125 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp"] Nov 01 00:45:54.133115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.133098 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.214599 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214561 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.214599 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214600 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.214799 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214627 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.214799 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214698 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22a6be0-4627-418a-aef8-38900f0445d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.214799 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214740 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.214799 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214760 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.214931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214827 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.214931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214883 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef75179e-d5ea-4254-a616-6f2f271728d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.214931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214901 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.215023 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214937 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwkp\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-kube-api-access-pzwkp\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.215023 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.214976 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.215095 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.215064 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.215131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.215096 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.215131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.215117 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22a6be0-4627-418a-aef8-38900f0445d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.215231 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.215207 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.215265 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.215246 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.215319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.215274 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b49w\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-kube-api-access-7b49w\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.215398 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.215376 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.293851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.293813 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94"] Nov 01 00:45:54.316437 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316404 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b49w\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-kube-api-access-7b49w\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.316622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316459 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.316622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316556 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.316622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316588 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.316622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316622 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.316857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316659 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22a6be0-4627-418a-aef8-38900f0445d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.316857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316718 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.316857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316746 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.316857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316776 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.316857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316841 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef75179e-d5ea-4254-a616-6f2f271728d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.317115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316869 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.317115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316924 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwkp\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-kube-api-access-pzwkp\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.317115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316948 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.317115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316977 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.317115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.317026 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.317115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.317028 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.317115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.317104 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22a6be0-4627-418a-aef8-38900f0445d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.317492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.317160 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.317492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.317196 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.317601 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.317563 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.317601 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.317587 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef75179e-d5ea-4254-a616-6f2f271728d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.317825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.317803 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.319105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.318057 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.319105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.318149 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.319334 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.316976 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.319334 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.318317 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.319334 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.318553 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.319334 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.319050 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22a6be0-4627-418a-aef8-38900f0445d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.320056 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.320007 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22a6be0-4627-418a-aef8-38900f0445d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.321225 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.321174 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.322238 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.322210 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.322946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.322921 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.379001 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.378920 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.379001 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.378950 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b49w\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-kube-api-access-7b49w\") pod \"maas-default-gateway-openshift-default-67bf986f99-v8p94\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.379395 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.379374 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwkp\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-kube-api-access-pzwkp\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.379576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.379559 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6ljpp\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.399612 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.399381 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:54.426279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.426246 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp"] Nov 01 00:45:54.446615 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.445766 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:54.836410 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.836385 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94"] Nov 01 00:45:54.837900 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:45:54.837875 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef75179e_d5ea_4254_a616_6f2f271728d2.slice/crio-62ceb80e332b5ab3ff60d525ee28e1ac124a0b8f631d79d3f4d4396eecb48c00 WatchSource:0}: Error finding container 62ceb80e332b5ab3ff60d525ee28e1ac124a0b8f631d79d3f4d4396eecb48c00: Status 404 returned error can't find the container with id 62ceb80e332b5ab3ff60d525ee28e1ac124a0b8f631d79d3f4d4396eecb48c00 Nov 01 00:45:54.840483 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.840444 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:54.840575 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.840519 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:54.840575 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.840553 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:54.890213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.890192 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp"] Nov 01 00:45:54.896962 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.896928 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:54.897046 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.897018 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:54.897094 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:54.897047 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:55.603819 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:55.603785 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" event={"ID":"ef75179e-d5ea-4254-a616-6f2f271728d2","Type":"ContainerStarted","Data":"eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5"} Nov 01 00:45:55.604211 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:55.603848 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" event={"ID":"ef75179e-d5ea-4254-a616-6f2f271728d2","Type":"ContainerStarted","Data":"62ceb80e332b5ab3ff60d525ee28e1ac124a0b8f631d79d3f4d4396eecb48c00"} Nov 01 00:45:55.605466 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:55.605439 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" event={"ID":"e22a6be0-4627-418a-aef8-38900f0445d7","Type":"ContainerStarted","Data":"5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd"} Nov 01 00:45:55.605602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:55.605485 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" event={"ID":"e22a6be0-4627-418a-aef8-38900f0445d7","Type":"ContainerStarted","Data":"1bf4c569cdcbfa7a9b580006e6b9e6cc25cf1588b08c285cf6d144a5fa86e5aa"} Nov 01 00:45:55.744039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:55.743964 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podStartSLOduration=1.743950696 podStartE2EDuration="1.743950696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:55.743669193 +0000 UTC m=+1300.618890222" watchObservedRunningTime="2026-11-01 00:45:55.743950696 +0000 UTC m=+1300.619171710" Nov 01 00:45:55.878386 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:55.876452 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podStartSLOduration=1.876428547 podStartE2EDuration="1.876428547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:55.874798529 +0000 UTC m=+1300.750019584" watchObservedRunningTime="2026-11-01 00:45:55.876428547 +0000 UTC m=+1300.751649589" Nov 01 00:45:56.399879 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:56.399506 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:45:56.401241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:56.400973 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v8p94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 00:45:56.401241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:56.401017 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 00:45:56.467312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:56.466346 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:45:56.471310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:56.467884 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6ljpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 00:45:56.471310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:56.467941 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 00:45:57.400738 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:57.400701 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v8p94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 00:45:57.401384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:57.401353 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 00:45:57.449752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:57.449714 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6ljpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 00:45:57.450037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:57.450006 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 00:45:57.451665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:57.451637 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:45:58.399888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:58.399823 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v8p94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 00:45:58.400079 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:58.399904 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 00:45:58.454800 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:58.454761 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6ljpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 00:45:58.455454 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:58.455420 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 00:45:59.399850 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:59.399805 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v8p94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 00:45:59.400044 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:59.399872 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 00:45:59.450314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:59.449681 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6ljpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 00:45:59.450314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:45:59.449778 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 00:46:00.403311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:00.402826 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v8p94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 00:46:00.403311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:00.402909 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 00:46:00.449312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:00.447779 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6ljpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 00:46:00.449312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:00.447874 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 00:46:01.404365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:01.401748 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v8p94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 00:46:01.404365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:01.401809 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 00:46:01.454313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:01.453492 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6ljpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 00:46:01.454313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:01.453565 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 00:46:02.400013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:02.399974 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-v8p94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 00:46:02.400215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:02.400043 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 00:46:02.446687 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:02.446646 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6ljpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 00:46:02.447180 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:02.446703 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 00:46:02.581162 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:02.581125 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:03.916071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:03.916038 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:46:03.916585 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:03.916505 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:46:03.917425 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:03.917403 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:46:03.937169 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:03.937141 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:46:03.937549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:03.937521 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:46:03.938264 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:03.938245 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:46:04.023617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.023583 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk"] Nov 01 00:46:04.027570 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.027549 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.060253 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.060221 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk"] Nov 01 00:46:04.125466 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125433 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.125620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125480 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.125620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125547 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.125620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125577 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ca2ced-3dc1-479d-ace9-4438014a6952-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.125620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125612 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.125797 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125633 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.125797 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125756 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgx6\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-kube-api-access-2wgx6\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.125870 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125795 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.125907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.125869 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227136 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227050 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227136 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227108 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227136 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227135 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ca2ced-3dc1-479d-ace9-4438014a6952-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227182 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227212 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227331 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgx6\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-kube-api-access-2wgx6\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227386 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227447 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227493 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227515 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.227833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.227798 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.228336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.228307 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.229148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.229124 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ca2ced-3dc1-479d-ace9-4438014a6952-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.230830 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.230806 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.245963 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.245924 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.246436 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.246412 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.291456 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.291426 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.291702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.291678 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgx6\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-kube-api-access-2wgx6\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.342420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.342389 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:04.774225 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.774200 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk"] Nov 01 00:46:04.775830 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:46:04.775806 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ca2ced_3dc1_479d_ace9_4438014a6952.slice/crio-f45d677e8e6809b6b31109fd967fff4d2ea7a7b7bd245dd8802be78fe9116a04 WatchSource:0}: Error finding container f45d677e8e6809b6b31109fd967fff4d2ea7a7b7bd245dd8802be78fe9116a04: Status 404 returned error can't find the container with id f45d677e8e6809b6b31109fd967fff4d2ea7a7b7bd245dd8802be78fe9116a04 Nov 01 00:46:04.778209 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.778175 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:04.778326 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.778242 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:04.778326 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:04.778274 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:05.203397 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.203363 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q"] Nov 01 00:46:05.207420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.207396 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242033 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242092 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242125 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242162 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242249 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txsf\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-kube-api-access-2txsf\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242397 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242499 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242529 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.243316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.242552 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.272312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.267366 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4143b7-openshift-default-dockercfg-fhj7l\"" Nov 01 00:46:05.279880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.279845 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q"] Nov 01 00:46:05.343951 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.343919 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344014 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344034 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344050 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344081 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344106 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344468 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344133 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344468 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344176 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344468 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344315 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txsf\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-kube-api-access-2txsf\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344468 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344435 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344468 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344490 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344834 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344772 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.344914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.344895 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.368314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.365088 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.376317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.372691 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.406700 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.406663 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txsf\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-kube-api-access-2txsf\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.406832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.406665 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.521753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.521663 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:05.706400 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.706363 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" event={"ID":"00ca2ced-3dc1-479d-ace9-4438014a6952","Type":"ContainerStarted","Data":"169ed7395600bafd45344b2ef447d7d943c41b830f1e614fc0b7707fa39d4b01"} Nov 01 00:46:05.706596 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.706432 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" event={"ID":"00ca2ced-3dc1-479d-ace9-4438014a6952","Type":"ContainerStarted","Data":"f45d677e8e6809b6b31109fd967fff4d2ea7a7b7bd245dd8802be78fe9116a04"} Nov 01 00:46:05.841099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:05.841042 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podStartSLOduration=2.841027194 podStartE2EDuration="2.841027194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:05.839475731 +0000 UTC m=+1310.714696761" watchObservedRunningTime="2026-11-01 00:46:05.841027194 +0000 UTC m=+1310.716248210" Nov 01 00:46:06.007576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.007547 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q"] Nov 01 00:46:06.009163 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:46:06.009131 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2d1e3f_03be_449c_be85_7b2f090b98f4.slice/crio-9207065b386706f812f63a15114f733d1262d150ec145b9fd6edd38dffa0823c WatchSource:0}: Error finding container 9207065b386706f812f63a15114f733d1262d150ec145b9fd6edd38dffa0823c: Status 404 returned error can't find the container with id 9207065b386706f812f63a15114f733d1262d150ec145b9fd6edd38dffa0823c Nov 01 00:46:06.011765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.011725 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:06.011874 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.011807 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:06.011874 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.011854 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:06.343071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.343031 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:06.344832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.344807 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 00:46:06.344939 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.344856 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 00:46:06.717846 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.717804 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" event={"ID":"0a2d1e3f-03be-449c-be85-7b2f090b98f4","Type":"ContainerStarted","Data":"8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d"} Nov 01 00:46:06.718024 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.717869 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" event={"ID":"0a2d1e3f-03be-449c-be85-7b2f090b98f4","Type":"ContainerStarted","Data":"9207065b386706f812f63a15114f733d1262d150ec145b9fd6edd38dffa0823c"} Nov 01 00:46:06.843468 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.843412 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podStartSLOduration=1.843394966 podStartE2EDuration="1.843394966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:06.841950446 +0000 UTC m=+1311.717171474" watchObservedRunningTime="2026-11-01 00:46:06.843394966 +0000 UTC m=+1311.718616071" Nov 01 00:46:06.971376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:06.970833 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:07.211365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:07.211316 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:07.342738 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:07.342708 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 00:46:07.342949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:07.342928 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 00:46:07.522179 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:07.522146 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:07.523915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:07.523886 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 00:46:07.524032 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:07.523934 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 00:46:07.809313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:07.796388 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:08.343372 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:08.343334 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 00:46:08.343535 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:08.343386 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 00:46:08.522140 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:08.522104 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 00:46:08.522351 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:08.522179 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 00:46:09.343930 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:09.343895 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 00:46:09.344084 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:09.343947 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 00:46:09.522435 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:09.522406 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 00:46:09.522886 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:09.522860 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 00:46:10.343640 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:10.343602 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 00:46:10.348595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:10.343673 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 00:46:10.522405 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:10.522374 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 00:46:10.522664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:10.522637 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 00:46:11.343703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:11.343656 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 00:46:11.343931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:11.343717 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 00:46:11.522392 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:11.522353 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 00:46:11.522603 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:11.522424 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 00:46:12.347645 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:12.347612 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:12.348175 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:12.348156 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:12.348885 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:12.348859 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:46:12.522464 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:12.522216 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 00:46:12.522464 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:12.522278 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 00:46:12.650236 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:12.650143 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:13.526959 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:13.526929 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:13.527416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:13.527200 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:13.528119 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:13.528100 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 00:46:17.103016 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:17.102976 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:17.559049 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:17.558993 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:19.490622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:19.490580 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:19.517733 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:46:19.517691 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:46:22.348885 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:22.348848 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:27.153325 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:27.153260 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:27.711787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:27.711748 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:31.947702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:31.947668 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:33.324663 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:33.324617 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:36.754140 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:36.754099 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:41.677579 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:41.677544 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:46.449753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:46.449716 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:49.504090 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:46:49.504050 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:46:50.695983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.695945 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz"] Nov 01 00:46:50.700320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.700299 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.758836 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.758805 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz"] Nov 01 00:46:50.764599 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.764572 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"distinct-w1-30fb19-kserve-self-signed-certs\"" Nov 01 00:46:50.800572 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.800524 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tls-certs\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.800572 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.800574 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkrz\" (UniqueName: \"kubernetes.io/projected/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-kube-api-access-nbkrz\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.800792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.800723 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tmp-dir\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.800792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.800764 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-model-cache\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.800925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.800908 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-home\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.801093 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.801071 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-dshm\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.851690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.851638 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:50.901970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.901931 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tls-certs\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.901970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.901968 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkrz\" (UniqueName: \"kubernetes.io/projected/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-kube-api-access-nbkrz\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.902241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.902036 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tmp-dir\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.902443 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.902422 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tmp-dir\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.902607 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.902434 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-model-cache\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.902737 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.902710 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-model-cache\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.903071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.903050 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-home\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.903368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.903350 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-dshm\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.903571 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.903430 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-home\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.905888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.905865 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tls-certs\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.906395 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.906349 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-dshm\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:50.965335 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:50.965226 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkrz\" (UniqueName: \"kubernetes.io/projected/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-kube-api-access-nbkrz\") pod \"distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:51.016127 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:51.016088 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:51.371883 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:51.371838 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:51.436446 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:51.436418 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz"] Nov 01 00:46:51.437950 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:46:51.437924 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4166a66b_53e0_4afa_adf6_02b33d0f8ec1.slice/crio-3134e70bdd9b2df3a211a012e727b77ac6f3e19fee639c549299243f3f2ab1ce WatchSource:0}: Error finding container 3134e70bdd9b2df3a211a012e727b77ac6f3e19fee639c549299243f3f2ab1ce: Status 404 returned error can't find the container with id 3134e70bdd9b2df3a211a012e727b77ac6f3e19fee639c549299243f3f2ab1ce Nov 01 00:46:52.034619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:52.034581 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" event={"ID":"4166a66b-53e0-4afa-adf6-02b33d0f8ec1","Type":"ContainerStarted","Data":"4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051"} Nov 01 00:46:52.034619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:52.034625 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" event={"ID":"4166a66b-53e0-4afa-adf6-02b33d0f8ec1","Type":"ContainerStarted","Data":"3134e70bdd9b2df3a211a012e727b77ac6f3e19fee639c549299243f3f2ab1ce"} Nov 01 00:46:52.156509 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:52.156442 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podStartSLOduration=2.156427749 podStartE2EDuration="2.156427749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:52.154366936 +0000 UTC m=+1357.029587951" watchObservedRunningTime="2026-11-01 00:46:52.156427749 +0000 UTC m=+1357.031648761" Nov 01 00:46:53.017008 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:53.016974 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:46:57.703208 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:57.703173 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:46:58.888830 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:46:58.888793 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:00.215257 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:00.215218 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:00.437223 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:00.437168 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:01.016767 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:01.016726 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:47:01.029336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:01.029276 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:47:01.056311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:01.053586 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:01.099098 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:01.099073 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:47:05.932122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:05.932070 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:08.151087 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.151052 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm"] Nov 01 00:47:08.155269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.155252 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.231759 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.231724 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm"] Nov 01 00:47:08.261522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.261496 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"distinct-w5-948e22-kserve-self-signed-certs\"" Nov 01 00:47:08.305623 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.305566 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:08.328867 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.328830 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r76g4\" (UniqueName: \"kubernetes.io/projected/f488eb5a-65f0-4b05-8f36-a76f84a1937d-kube-api-access-r76g4\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.329131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.328960 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-model-cache\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.329131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.329057 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-home\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.329131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.329104 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tmp-dir\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.329445 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.329180 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tls-certs\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.329445 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.329211 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-dshm\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.430704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.430617 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-home\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.430704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.430666 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tmp-dir\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.430914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.430715 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tls-certs\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.430914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.430735 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-dshm\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.430914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.430815 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r76g4\" (UniqueName: \"kubernetes.io/projected/f488eb5a-65f0-4b05-8f36-a76f84a1937d-kube-api-access-r76g4\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.430914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.430867 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-model-cache\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.431116 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.431084 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-home\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.431195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.431171 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tmp-dir\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.431251 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.431211 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-model-cache\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.433200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.433178 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-dshm\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.433472 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.433456 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tls-certs\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.494337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.494279 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r76g4\" (UniqueName: \"kubernetes.io/projected/f488eb5a-65f0-4b05-8f36-a76f84a1937d-kube-api-access-r76g4\") pod \"distinct-w5-948e22-kserve-d6fbc466b-vw7mm\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:08.768777 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:08.768691 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:09.156672 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:09.152079 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm"] Nov 01 00:47:09.156672 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:47:09.152791 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf488eb5a_65f0_4b05_8f36_a76f84a1937d.slice/crio-2b2382d0537feb851710d549368a6dc31dce2894547547af9db6ec09ab9d8760 WatchSource:0}: Error finding container 2b2382d0537feb851710d549368a6dc31dce2894547547af9db6ec09ab9d8760: Status 404 returned error can't find the container with id 2b2382d0537feb851710d549368a6dc31dce2894547547af9db6ec09ab9d8760 Nov 01 00:47:10.149057 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:10.149019 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" event={"ID":"f488eb5a-65f0-4b05-8f36-a76f84a1937d","Type":"ContainerStarted","Data":"3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded"} Nov 01 00:47:10.149304 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:10.149072 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" event={"ID":"f488eb5a-65f0-4b05-8f36-a76f84a1937d","Type":"ContainerStarted","Data":"2b2382d0537feb851710d549368a6dc31dce2894547547af9db6ec09ab9d8760"} Nov 01 00:47:10.270199 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:10.270150 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podStartSLOduration=2.270135234 podStartE2EDuration="2.270135234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:47:10.268118971 +0000 UTC m=+1375.143339985" watchObservedRunningTime="2026-11-01 00:47:10.270135234 +0000 UTC m=+1375.145356249" Nov 01 00:47:10.768949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:10.768914 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:10.777108 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:10.777071 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:13.584192 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:13.584159 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:15.656455 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:15.656418 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:18.768982 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:18.768945 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:18.783849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:18.783817 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:19.233093 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:19.233059 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:47:19.510723 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:47:19.510630 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:47:20.535020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:20.534988 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:25.488165 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:25.488123 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:28.075131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:28.075095 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:30.333881 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:30.333844 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:31.805014 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.804981 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82"] Nov 01 00:47:31.809990 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.809966 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.855651 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855616 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.855839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855663 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.855839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855697 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.855839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855716 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.855839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855731 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.855839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855792 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.856099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855844 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.856099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855884 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.856099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.855935 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7j7k\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-kube-api-access-m7j7k\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.920578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.920540 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82"] Nov 01 00:47:31.960100 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960059 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.960330 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960145 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.960330 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960230 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7j7k\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-kube-api-access-m7j7k\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.960330 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960325 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.960505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960376 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.960505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960421 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.960505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960461 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.960505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960485 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.960692 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.960578 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.961162 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.961132 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.961847 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.961822 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.962578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.962557 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.963043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.963014 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.963925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.963896 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.966549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.966515 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:31.968011 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:31.967986 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:32.023640 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:32.023604 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:32.023947 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:32.023927 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7j7k\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-kube-api-access-m7j7k\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:32.125961 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:32.125872 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:32.556814 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:32.556788 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82"] Nov 01 00:47:32.558020 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:47:32.557993 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb281633e_b211_4af1_b2d5_d5a700d1e9e9.slice/crio-508e1af718e094095244c889cfa6cd18c63da54c1e2719caf92d9e2f3a40c7c1 WatchSource:0}: Error finding container 508e1af718e094095244c889cfa6cd18c63da54c1e2719caf92d9e2f3a40c7c1: Status 404 returned error can't find the container with id 508e1af718e094095244c889cfa6cd18c63da54c1e2719caf92d9e2f3a40c7c1 Nov 01 00:47:32.560557 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:32.560524 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:32.560624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:32.560592 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:32.560624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:32.560620 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:33.304619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:33.304584 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" event={"ID":"b281633e-b211-4af1-b2d5-d5a700d1e9e9","Type":"ContainerStarted","Data":"df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750"} Nov 01 00:47:33.305037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:33.304639 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" event={"ID":"b281633e-b211-4af1-b2d5-d5a700d1e9e9","Type":"ContainerStarted","Data":"508e1af718e094095244c889cfa6cd18c63da54c1e2719caf92d9e2f3a40c7c1"} Nov 01 00:47:33.488391 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:33.488336 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podStartSLOduration=2.488321703 podStartE2EDuration="2.488321703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:47:33.487137928 +0000 UTC m=+1398.362358942" watchObservedRunningTime="2026-11-01 00:47:33.488321703 +0000 UTC m=+1398.363542715" Nov 01 00:47:34.126982 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:34.126953 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:34.128343 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:34.128317 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 00:47:34.128478 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:34.128376 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 00:47:35.127029 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:35.126996 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 00:47:35.127455 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:35.127057 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 00:47:35.184314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:35.181153 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:36.126979 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:36.126937 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 00:47:36.127156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:36.127012 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 00:47:37.126631 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:37.126596 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 00:47:37.126821 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:37.126662 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 00:47:38.126404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:38.126359 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 00:47:38.126897 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:38.126435 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 00:47:39.127021 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:39.126981 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 00:47:39.127457 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:39.127057 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 00:47:40.035479 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:40.035427 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:40.131114 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:40.131084 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:40.131566 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:40.131548 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:40.132335 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:40.132316 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 00:47:44.828731 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:44.828697 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:49.506176 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:47:49.506139 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:47:49.638683 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:49.638648 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:54.574717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:54.574670 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:47:59.646340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:47:59.646263 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:04.534768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:04.534731 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:07.643692 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.643651 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s"] Nov 01 00:48:07.647859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.647836 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.753755 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.753696 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"openshift-service-ca.crt\"" Nov 01 00:48:07.753956 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.753767 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"distinct-2-w3-5a9c19-kserve-self-signed-certs\"" Nov 01 00:48:07.753956 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.753771 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"kube-root-ca.crt\"" Nov 01 00:48:07.753956 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.753793 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"default-dockercfg-n9ktj\"" Nov 01 00:48:07.756332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.756309 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s"] Nov 01 00:48:07.757951 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.757932 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828bbae3-be6b-45df-8516-9f659098e61d-tls-certs\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.757998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.757978 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-dshm\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.758106 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.758091 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bgfm\" (UniqueName: \"kubernetes.io/projected/828bbae3-be6b-45df-8516-9f659098e61d-kube-api-access-7bgfm\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.758152 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.758140 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-model-cache\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.758191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.758170 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-home\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.758251 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.758234 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-tmp-dir\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.797826 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.797788 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:07.859856 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.859828 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828bbae3-be6b-45df-8516-9f659098e61d-tls-certs\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.860065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.859877 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-dshm\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.860065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.859907 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bgfm\" (UniqueName: \"kubernetes.io/projected/828bbae3-be6b-45df-8516-9f659098e61d-kube-api-access-7bgfm\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.860065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.859951 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-model-cache\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.860065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.859973 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-home\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.860065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.860009 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-tmp-dir\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.860431 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.860402 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-model-cache\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.860554 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.860477 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-home\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.860554 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.860520 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-tmp-dir\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.862389 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.862365 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-dshm\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.862710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.862693 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828bbae3-be6b-45df-8516-9f659098e61d-tls-certs\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.922265 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.922193 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bgfm\" (UniqueName: \"kubernetes.io/projected/828bbae3-be6b-45df-8516-9f659098e61d-kube-api-access-7bgfm\") pod \"distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:07.961340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:07.961272 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:08.364875 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:08.364855 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s"] Nov 01 00:48:08.366195 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:48:08.366168 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828bbae3_be6b_45df_8516_9f659098e61d.slice/crio-3133d1193a513ac5c536d64b2e6d841b060f5b2fce6bbc00c27014a0c6485db5 WatchSource:0}: Error finding container 3133d1193a513ac5c536d64b2e6d841b060f5b2fce6bbc00c27014a0c6485db5: Status 404 returned error can't find the container with id 3133d1193a513ac5c536d64b2e6d841b060f5b2fce6bbc00c27014a0c6485db5 Nov 01 00:48:08.525268 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:08.525232 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" event={"ID":"828bbae3-be6b-45df-8516-9f659098e61d","Type":"ContainerStarted","Data":"062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4"} Nov 01 00:48:08.525268 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:08.525277 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" event={"ID":"828bbae3-be6b-45df-8516-9f659098e61d","Type":"ContainerStarted","Data":"3133d1193a513ac5c536d64b2e6d841b060f5b2fce6bbc00c27014a0c6485db5"} Nov 01 00:48:08.646676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:08.646580 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podStartSLOduration=1.646564128 podStartE2EDuration="1.646564128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:48:08.644043397 +0000 UTC m=+1433.519264412" watchObservedRunningTime="2026-11-01 00:48:08.646564128 +0000 UTC m=+1433.521785160" Nov 01 00:48:08.961773 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:08.961690 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:09.482073 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:09.482034 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:14.327145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:14.327108 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:17.964070 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:17.964024 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:17.982754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:17.982709 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:18.600169 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:18.600135 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:48:19.256450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:19.256398 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:19.505032 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:48:19.504999 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:48:22.404705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:22.404676 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:22.623408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:22.623368 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:24.158439 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:24.158190 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:28.427330 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:28.427267 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:29.053401 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:29.053364 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:33.863206 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:33.863166 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:38.810908 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:38.810872 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:39.208880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:39.208792 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:43.851135 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:43.851087 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:48.662115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:48.662069 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:49.506182 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:48:49.506148 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:48:50.142310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.142261 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns"] Nov 01 00:48:50.146654 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.146637 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.200638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.200601 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"distinct-2-w0-32439f-kserve-self-signed-certs\"" Nov 01 00:48:50.202452 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.202417 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-dshm\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.202793 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.202770 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-model-cache\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.202975 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.202949 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tls-certs\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.203098 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.203085 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6skcr\" (UniqueName: \"kubernetes.io/projected/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-kube-api-access-6skcr\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.203489 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.203467 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-home\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.203600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.203585 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tmp-dir\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.228532 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.228500 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns"] Nov 01 00:48:50.304980 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.304941 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-home\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.305192 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.305037 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tmp-dir\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.305192 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.305091 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-dshm\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.305192 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.305112 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-model-cache\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.305192 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.305136 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tls-certs\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.305983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.305254 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6skcr\" (UniqueName: \"kubernetes.io/projected/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-kube-api-access-6skcr\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.305983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.305609 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-home\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.306761 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.306735 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-model-cache\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.307116 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.307067 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tmp-dir\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.312215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.312185 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-dshm\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.312981 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.312921 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tls-certs\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.328707 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.328655 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:50.383459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.383427 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6skcr\" (UniqueName: \"kubernetes.io/projected/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-kube-api-access-6skcr\") pod \"distinct-2-w0-32439f-kserve-79f769c4d5-zzgns\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.458995 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.458910 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:50.842337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:50.842313 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns"] Nov 01 00:48:50.842999 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:48:50.842975 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d5018c8_eafb_4c4f_87a5_9035c638eaf1.slice/crio-5bc6a03b29907a695800556c75d03b798da1006fcfb2632a8034e9e4603a3a89 WatchSource:0}: Error finding container 5bc6a03b29907a695800556c75d03b798da1006fcfb2632a8034e9e4603a3a89: Status 404 returned error can't find the container with id 5bc6a03b29907a695800556c75d03b798da1006fcfb2632a8034e9e4603a3a89 Nov 01 00:48:51.787679 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:51.787643 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" event={"ID":"6d5018c8-eafb-4c4f-87a5-9035c638eaf1","Type":"ContainerStarted","Data":"6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec"} Nov 01 00:48:51.787679 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:51.787686 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" event={"ID":"6d5018c8-eafb-4c4f-87a5-9035c638eaf1","Type":"ContainerStarted","Data":"5bc6a03b29907a695800556c75d03b798da1006fcfb2632a8034e9e4603a3a89"} Nov 01 00:48:51.918915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:51.918854 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podStartSLOduration=1.918837111 podStartE2EDuration="1.918837111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:48:51.916604906 +0000 UTC m=+1476.791825920" watchObservedRunningTime="2026-11-01 00:48:51.918837111 +0000 UTC m=+1476.794058136" Nov 01 00:48:52.459338 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:52.459304 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:48:53.648969 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:53.648940 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:57.576122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:57.576078 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:48:59.645071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:48:59.645038 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:00.459264 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:00.459227 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:49:00.472669 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:00.472644 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:49:00.868665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:00.868638 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:49:03.864904 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:03.864843 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:05.124744 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.124709 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9"] Nov 01 00:49:05.128797 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.128773 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.212327 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.212266 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-dshm\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.212523 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.212445 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tls-certs\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.212641 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.212622 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmjx\" (UniqueName: \"kubernetes.io/projected/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-kube-api-access-bzmjx\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.212711 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.212695 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-model-cache\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.212790 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.212776 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-home\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.212832 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.212803 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tmp-dir\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.235266 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.235230 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-30fb19\"/\"distinct-2-w1-30fb19-kserve-self-signed-certs\"" Nov 01 00:49:05.237720 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.237698 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9"] Nov 01 00:49:05.302057 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.302023 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:05.314500 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.314467 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-home\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.314500 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.314504 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tmp-dir\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.314740 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.314551 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-dshm\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.314740 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.314598 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tls-certs\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.314740 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.314719 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmjx\" (UniqueName: \"kubernetes.io/projected/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-kube-api-access-bzmjx\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.314899 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.314782 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-model-cache\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.314953 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.314909 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-home\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.315106 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.315083 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tmp-dir\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.315273 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.315246 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-model-cache\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.317825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.317798 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-dshm\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.318088 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.318070 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tls-certs\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.378658 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.378575 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmjx\" (UniqueName: \"kubernetes.io/projected/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-kube-api-access-bzmjx\") pod \"distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.441947 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.441914 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:05.828348 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:49:05.828248 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf73da4f_eab7_4f6b_9ecd_18e5d2d1367d.slice/crio-1dd527009ddff51def3ec6e4ef9b69c87d12cc3d70f71ac76176a7235b6fb410 WatchSource:0}: Error finding container 1dd527009ddff51def3ec6e4ef9b69c87d12cc3d70f71ac76176a7235b6fb410: Status 404 returned error can't find the container with id 1dd527009ddff51def3ec6e4ef9b69c87d12cc3d70f71ac76176a7235b6fb410 Nov 01 00:49:05.828729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.828704 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9"] Nov 01 00:49:05.892152 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:05.892122 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" event={"ID":"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d","Type":"ContainerStarted","Data":"1dd527009ddff51def3ec6e4ef9b69c87d12cc3d70f71ac76176a7235b6fb410"} Nov 01 00:49:06.902148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:06.902095 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" event={"ID":"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d","Type":"ContainerStarted","Data":"26a81659fb46a8b41b87f5769c54badb5633e94f0dd6ba33d9ef79ea7ef0f2a5"} Nov 01 00:49:07.024078 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:07.024019 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podStartSLOduration=2.024003644 podStartE2EDuration="2.024003644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:07.022036995 +0000 UTC m=+1491.897258009" watchObservedRunningTime="2026-11-01 00:49:07.024003644 +0000 UTC m=+1491.899224659" Nov 01 00:49:07.442705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:07.442663 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:08.766154 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:08.766100 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:13.477834 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:13.477785 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:13.700259 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:13.700224 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:15.442080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:15.442045 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:15.460925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:15.460898 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:15.969107 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:15.969079 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:49:16.812195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.812165 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr"] Nov 01 00:49:16.816619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.816592 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:16.874387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.874355 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr"] Nov 01 00:49:16.922387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.922353 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-dshm\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:16.922387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.922386 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-model-cache\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:16.922598 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.922428 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"distinct-2-w5-948e22-kserve-self-signed-certs\"" Nov 01 00:49:16.922598 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.922469 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tmp-dir\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:16.922598 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.922519 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvsd4\" (UniqueName: \"kubernetes.io/projected/6499de7b-330b-48bf-ae0a-86d6f1882c8c-kube-api-access-kvsd4\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:16.922598 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.922595 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-home\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:16.922730 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.922637 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tls-certs\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:16.972015 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:16.971979 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:17.024224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024183 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tls-certs\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.024450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024355 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-dshm\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.024450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024375 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-model-cache\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.024450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024416 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tmp-dir\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.024635 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024469 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvsd4\" (UniqueName: \"kubernetes.io/projected/6499de7b-330b-48bf-ae0a-86d6f1882c8c-kube-api-access-kvsd4\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.024635 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024534 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-home\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.024892 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024849 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-model-cache\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.024892 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024877 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tmp-dir\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.025013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.024919 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-home\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.026771 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.026748 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-dshm\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.027060 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.027042 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tls-certs\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.098960 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.098877 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvsd4\" (UniqueName: \"kubernetes.io/projected/6499de7b-330b-48bf-ae0a-86d6f1882c8c-kube-api-access-kvsd4\") pod \"distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.135595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.135561 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:17.524907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.524879 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr"] Nov 01 00:49:17.526580 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:49:17.526549 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6499de7b_330b_48bf_ae0a_86d6f1882c8c.slice/crio-f6f199264cfaa2d4bd8f64acb587dd52154028c6d67f8c48bd233c047039c294 WatchSource:0}: Error finding container f6f199264cfaa2d4bd8f64acb587dd52154028c6d67f8c48bd233c047039c294: Status 404 returned error can't find the container with id f6f199264cfaa2d4bd8f64acb587dd52154028c6d67f8c48bd233c047039c294 Nov 01 00:49:17.976513 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.976478 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" event={"ID":"6499de7b-330b-48bf-ae0a-86d6f1882c8c","Type":"ContainerStarted","Data":"52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554"} Nov 01 00:49:17.976925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:17.976528 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" event={"ID":"6499de7b-330b-48bf-ae0a-86d6f1882c8c","Type":"ContainerStarted","Data":"f6f199264cfaa2d4bd8f64acb587dd52154028c6d67f8c48bd233c047039c294"} Nov 01 00:49:18.102361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:18.102307 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podStartSLOduration=2.102267888 podStartE2EDuration="2.102267888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:18.101948976 +0000 UTC m=+1502.977169992" watchObservedRunningTime="2026-11-01 00:49:18.102267888 +0000 UTC m=+1502.977488949" Nov 01 00:49:18.135851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:18.135820 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:18.634362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:18.634328 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:19.506080 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:49:19.506041 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:49:21.057072 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.057033 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk"] Nov 01 00:49:21.061437 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.061417 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.115704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.115672 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-32439f-openshift-default-dockercfg-n9rr6\"" Nov 01 00:49:21.170880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.170841 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk"] Nov 01 00:49:21.184920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.184868 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.185129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.184927 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.185129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.185009 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvx7\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-kube-api-access-fcvx7\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.185258 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.185123 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.185258 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.185151 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.185258 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.185179 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.185413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.185318 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9027416f-3c14-453c-ae29-81c0c7819e2d-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.185413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.185378 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.185550 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.185537 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.286809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.286774 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.286903 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.286971 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.286991 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287008 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvx7\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-kube-api-access-fcvx7\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287061 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287088 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287117 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287157 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9027416f-3c14-453c-ae29-81c0c7819e2d-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287538 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287513 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287585 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287523 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287810 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287787 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.287877 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.287831 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.288099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.288076 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9027416f-3c14-453c-ae29-81c0c7819e2d-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.289489 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.289470 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.289599 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.289576 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.350709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.350636 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.350853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.350766 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvx7\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-kube-api-access-fcvx7\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.377182 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.377148 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:21.811848 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.811817 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk"] Nov 01 00:49:21.812558 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:49:21.812524 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9027416f_3c14_453c_ae29_81c0c7819e2d.slice/crio-3ec9f7605f02739daace9d39b0819d3c84ff84a2e6e65276f3a84f6df5061521 WatchSource:0}: Error finding container 3ec9f7605f02739daace9d39b0819d3c84ff84a2e6e65276f3a84f6df5061521: Status 404 returned error can't find the container with id 3ec9f7605f02739daace9d39b0819d3c84ff84a2e6e65276f3a84f6df5061521 Nov 01 00:49:21.815151 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.815117 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:21.815235 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.815180 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:21.815235 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:21.815210 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:22.003375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:22.003338 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" event={"ID":"9027416f-3c14-453c-ae29-81c0c7819e2d","Type":"ContainerStarted","Data":"a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91"} Nov 01 00:49:22.003557 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:22.003396 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" event={"ID":"9027416f-3c14-453c-ae29-81c0c7819e2d","Type":"ContainerStarted","Data":"3ec9f7605f02739daace9d39b0819d3c84ff84a2e6e65276f3a84f6df5061521"} Nov 01 00:49:22.135972 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:22.135837 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podStartSLOduration=1.135812095 podStartE2EDuration="1.135812095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:22.131780522 +0000 UTC m=+1507.007001548" watchObservedRunningTime="2026-11-01 00:49:22.135812095 +0000 UTC m=+1507.011033111" Nov 01 00:49:22.378386 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:22.378350 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:22.379990 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:22.379963 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 00:49:22.380064 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:22.380017 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 00:49:23.378071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:23.378032 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 00:49:23.378537 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:23.378090 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 00:49:23.464524 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:23.463920 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:24.377893 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:24.377858 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 00:49:24.378099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:24.377926 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 00:49:25.378033 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:25.378003 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 00:49:25.378208 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:25.378056 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 00:49:26.378710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:26.378666 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 00:49:26.379146 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:26.378750 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 00:49:27.136213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:27.136171 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:27.172026 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:27.171994 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:27.377654 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:27.377606 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 00:49:27.377877 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:27.377688 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 00:49:28.053767 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:28.053735 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:49:28.378583 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:28.378488 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 00:49:28.378727 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:28.378574 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 00:49:28.557994 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:28.557940 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:28.830228 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:28.830188 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:29.382433 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:29.382403 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:29.382928 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:29.382817 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:29.383565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:29.383544 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 00:49:33.855088 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:33.847655 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:49:33.855682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:33.855400 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:49:34.036315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:34.029425 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:34.316665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:34.315450 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:49:34.483769 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:34.483407 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:49:36.664046 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:36.664004 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:39.476440 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:39.476395 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:44.448498 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:44.448459 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:49.349550 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:49.349515 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:49.506595 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:49:49.506560 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:49:51.990883 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:51.990848 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk"] Nov 01 00:49:51.995580 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:51.995556 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.060681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.060644 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.060863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.060705 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.060863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.060827 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.061016 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.060991 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.061102 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.061083 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.061227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.061183 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.061280 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.061251 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.061418 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.061398 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fffr\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-kube-api-access-4fffr\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.061485 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.061443 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.162733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.162664 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.162733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.162732 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163012 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.162759 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163012 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.162836 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163012 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.162931 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163012 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.162969 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163012 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.163006 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.163078 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fffr\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-kube-api-access-4fffr\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.163108 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163481 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.163424 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163611 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.163594 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.163657 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.163752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.163704 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.164166 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.164139 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.169326 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.166831 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.169326 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.167194 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.209641 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.209603 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk"] Nov 01 00:49:52.227553 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.227513 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.227742 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.227581 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fffr\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-kube-api-access-4fffr\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.290743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.290711 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw"] Nov 01 00:49:52.296049 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.296025 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.310054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.310028 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:52.356158 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.356098 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw"] Nov 01 00:49:52.366654 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.366491 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e14247-ec2a-481c-a38a-e4030afb6257-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.366654 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.366563 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.367695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.366986 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.367695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.367046 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.367695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.367083 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrbk\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-kube-api-access-jxrbk\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.367695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.367119 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.367695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.367161 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.367695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.367376 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.367695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.367415 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469173 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469133 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e14247-ec2a-481c-a38a-e4030afb6257-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469173 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469174 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469261 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469317 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469356 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrbk\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-kube-api-access-jxrbk\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469389 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469426 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469598 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469630 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469867 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469833 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.469959 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469930 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.470018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.469960 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.470153 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.470133 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e14247-ec2a-481c-a38a-e4030afb6257-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.470194 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.470133 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.471883 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.471864 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.471998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.471984 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.532210 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.532172 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.532384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.532299 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrbk\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-kube-api-access-jxrbk\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.613840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.613746 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:52.769877 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.769851 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk"] Nov 01 00:49:52.770956 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:49:52.770929 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7626ba6_6131_4a6a_ba8e_08531f8a1daf.slice/crio-ed9927e019f5c5b633b87cc331a1ab26023dba5e3d8d73a9ea393e918837d3a7 WatchSource:0}: Error finding container ed9927e019f5c5b633b87cc331a1ab26023dba5e3d8d73a9ea393e918837d3a7: Status 404 returned error can't find the container with id ed9927e019f5c5b633b87cc331a1ab26023dba5e3d8d73a9ea393e918837d3a7 Nov 01 00:49:52.773830 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.773797 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:52.773933 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.773879 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:52.773997 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.773930 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:52.997179 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:52.997150 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw"] Nov 01 00:49:52.997862 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:49:52.997837 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e14247_ec2a_481c_a38a_e4030afb6257.slice/crio-dfafa80847a3c9a2d84f5169073602a7b2f68f26b5a1b3828107d7eeac4647a7 WatchSource:0}: Error finding container dfafa80847a3c9a2d84f5169073602a7b2f68f26b5a1b3828107d7eeac4647a7: Status 404 returned error can't find the container with id dfafa80847a3c9a2d84f5169073602a7b2f68f26b5a1b3828107d7eeac4647a7 Nov 01 00:49:53.000609 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.000572 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:53.000726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.000659 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:53.000726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.000709 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:53.314752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.314704 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" event={"ID":"a4e14247-ec2a-481c-a38a-e4030afb6257","Type":"ContainerStarted","Data":"03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a"} Nov 01 00:49:53.314968 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.314776 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" event={"ID":"a4e14247-ec2a-481c-a38a-e4030afb6257","Type":"ContainerStarted","Data":"dfafa80847a3c9a2d84f5169073602a7b2f68f26b5a1b3828107d7eeac4647a7"} Nov 01 00:49:53.316896 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.316864 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" event={"ID":"e7626ba6-6131-4a6a-ba8e-08531f8a1daf","Type":"ContainerStarted","Data":"2d81f6741fca5cdde8fd998bb97073d11623e2fe13d5588f2fe26c76cc250d54"} Nov 01 00:49:53.317015 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.316925 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" event={"ID":"e7626ba6-6131-4a6a-ba8e-08531f8a1daf","Type":"ContainerStarted","Data":"ed9927e019f5c5b633b87cc331a1ab26023dba5e3d8d73a9ea393e918837d3a7"} Nov 01 00:49:53.439393 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.439328 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podStartSLOduration=1.439309505 podStartE2EDuration="1.439309505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:53.438116929 +0000 UTC m=+1538.313337958" watchObservedRunningTime="2026-11-01 00:49:53.439309505 +0000 UTC m=+1538.314530521" Nov 01 00:49:53.613905 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:53.613809 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:49:54.311040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:54.310999 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:49:54.312255 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:54.312229 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:49:54.312375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:54.312280 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 00:49:54.420188 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:54.416838 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:54.615482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:54.615387 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 00:49:54.615752 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:54.615697 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 00:49:55.311039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:55.310991 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:49:55.311261 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:55.311071 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 00:49:55.615027 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:55.614931 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 00:49:55.615027 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:55.614993 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 00:49:56.311938 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:56.311891 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:49:56.312412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:56.311954 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 00:49:56.614430 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:56.614335 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 00:49:56.614430 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:56.614392 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 00:49:57.311152 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:57.311115 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:49:57.311362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:57.311170 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 00:49:57.614805 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:57.614704 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 00:49:57.614805 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:57.614772 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 00:49:58.310918 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:58.310886 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:49:58.311110 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:58.310932 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 00:49:58.615135 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:58.615047 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 00:49:58.615135 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:58.615105 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 00:49:59.310966 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:59.310928 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:49:59.311147 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:59.310998 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 00:49:59.384359 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:59.384319 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:49:59.614362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:59.614234 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 00:49:59.614539 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:49:59.614381 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 00:50:00.317358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:00.317327 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:50:00.317887 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:00.317814 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:50:00.318705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:00.318679 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:50:00.492705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:00.492638 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podStartSLOduration=9.492617315 podStartE2EDuration="9.492617315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:53.574042208 +0000 UTC m=+1538.449263236" watchObservedRunningTime="2026-11-01 00:50:00.492617315 +0000 UTC m=+1545.367838331" Nov 01 00:50:00.621522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:00.621438 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:50:00.622496 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:00.622471 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:50:00.623279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:00.623256 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:50:02.606453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.606416 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl"] Nov 01 00:50:02.611316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.611262 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.657177 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.657144 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k"] Nov 01 00:50:02.661447 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.661426 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.674172 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674141 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.674360 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674190 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.674436 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674421 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.674495 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674469 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.674642 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674619 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwpl\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-kube-api-access-czwpl\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.674697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674679 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.674754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674737 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.674793 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674766 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.674839 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.674793 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.715947 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.715908 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl"] Nov 01 00:50:02.737804 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.737772 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k"] Nov 01 00:50:02.776459 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776429 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.776646 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776498 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.776687 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776672 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.776754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776738 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.776809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776800 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.776846 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776835 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45594\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-kube-api-access-45594\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.776892 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776869 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.776943 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776928 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.776984 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.776939 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777022 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777012 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwpl\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-kube-api-access-czwpl\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777059 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777046 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.777114 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777097 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777165 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777150 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777210 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777183 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777210 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777185 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777233 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.777321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777262 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777407 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777357 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.777452 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777416 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.777452 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777423 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777659 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777642 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.777977 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.777954 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.778523 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.778496 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.779915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.779893 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.780715 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.780697 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.840523 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.840490 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.840672 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.840497 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwpl\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-kube-api-access-czwpl\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.879411 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879329 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45594\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-kube-api-access-45594\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879411 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879366 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879411 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879409 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879453 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879546 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879613 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879666 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879762 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879830 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.879924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879875 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.880113 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.879828 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.880338 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.880315 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.880421 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.880343 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.880991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.880971 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.882428 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.882407 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.882520 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.882467 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.927175 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.927138 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:02.942008 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.941980 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45594\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-kube-api-access-45594\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.942931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.942909 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:02.976265 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:02.976222 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:03.367718 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.367683 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl"] Nov 01 00:50:03.369512 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:50:03.369486 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123f4e51_b22a_4a7b_bc92_f3fefd1b2f3f.slice/crio-b88d160eb1f44b58727538a6d18b371f28cc5d6355256561b87398d4fb5a5248 WatchSource:0}: Error finding container b88d160eb1f44b58727538a6d18b371f28cc5d6355256561b87398d4fb5a5248: Status 404 returned error can't find the container with id b88d160eb1f44b58727538a6d18b371f28cc5d6355256561b87398d4fb5a5248 Nov 01 00:50:03.372554 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.372510 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.372642 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.372608 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.372679 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.372651 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.420956 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.420930 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k"] Nov 01 00:50:03.424534 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:50:03.424502 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e9e512_7cdd_4952_8752_edec0ca50c4f.slice/crio-5404ed61bf842a9cc202d1619370b8899aaec4b36738ff39b3c35d16bf021325 WatchSource:0}: Error finding container 5404ed61bf842a9cc202d1619370b8899aaec4b36738ff39b3c35d16bf021325: Status 404 returned error can't find the container with id 5404ed61bf842a9cc202d1619370b8899aaec4b36738ff39b3c35d16bf021325 Nov 01 00:50:03.427433 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.427400 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.427519 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.427465 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.427519 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.427493 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.434423 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.434393 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" event={"ID":"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f","Type":"ContainerStarted","Data":"b88d160eb1f44b58727538a6d18b371f28cc5d6355256561b87398d4fb5a5248"} Nov 01 00:50:03.435970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:03.435943 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" event={"ID":"d1e9e512-7cdd-4952-8752-edec0ca50c4f","Type":"ContainerStarted","Data":"5404ed61bf842a9cc202d1619370b8899aaec4b36738ff39b3c35d16bf021325"} Nov 01 00:50:04.359312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.351391 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:04.472324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.469356 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" event={"ID":"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f","Type":"ContainerStarted","Data":"004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c"} Nov 01 00:50:04.479334 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.472987 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" event={"ID":"d1e9e512-7cdd-4952-8752-edec0ca50c4f","Type":"ContainerStarted","Data":"854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458"} Nov 01 00:50:04.658435 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.658312 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podStartSLOduration=2.658273484 podStartE2EDuration="2.658273484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:04.656876008 +0000 UTC m=+1549.532097021" watchObservedRunningTime="2026-11-01 00:50:04.658273484 +0000 UTC m=+1549.533494501" Nov 01 00:50:04.798390 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.789506 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podStartSLOduration=2.789485312 podStartE2EDuration="2.789485312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:04.787759446 +0000 UTC m=+1549.662980462" watchObservedRunningTime="2026-11-01 00:50:04.789485312 +0000 UTC m=+1549.664706328" Nov 01 00:50:04.928325 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.928214 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:04.929720 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.929695 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 00:50:04.929858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.929743 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 00:50:04.977312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.976679 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:04.996313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.987958 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:50:04.996313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:04.988029 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 00:50:05.903419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.902593 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt"] Nov 01 00:50:05.910417 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.910389 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.922748 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.922723 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq"] Nov 01 00:50:05.927496 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.927471 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 00:50:05.927624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.927514 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 00:50:05.927931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.927915 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt"] Nov 01 00:50:05.928043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.928029 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:05.928787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.928771 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.928831 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.928817 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.928874 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.928848 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.928964 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.928944 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.929086 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.929072 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.929127 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.929101 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f86m7\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-kube-api-access-f86m7\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.929181 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.929133 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.929298 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.929270 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.929353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.929342 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:05.976745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.976709 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:50:05.977063 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:05.977022 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 00:50:06.018746 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.018691 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq"] Nov 01 00:50:06.030360 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.030325 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvt74\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-kube-api-access-cvt74\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.030532 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.030383 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.030532 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.030422 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f86m7\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-kube-api-access-f86m7\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.030532 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.030473 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.030532 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.030500 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.030909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.030693 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.030909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.030730 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.030909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.030849 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.031134 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031029 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.031134 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031106 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.031247 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031134 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.031247 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031176 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.031247 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031203 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.031453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031255 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.031453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031270 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.031453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031321 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.031453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031433 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.031676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.031474 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.034105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.032005 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.034105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.032203 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.034105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.032547 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.034105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.032724 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.034105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.032851 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.036019 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.035999 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.036715 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.036666 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.107340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.101153 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.117040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.116916 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f86m7\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-kube-api-access-f86m7\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.132622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.132588 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.133243 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.133219 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.133578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.133560 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.133707 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.133692 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.133833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.133818 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvt74\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-kube-api-access-cvt74\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.133982 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.133965 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.134175 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.134160 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.134469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.134450 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.134617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.134598 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.134743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.134716 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.135350 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.135002 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.135350 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.135190 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.135350 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.135263 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.135568 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.135457 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.136565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.136078 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.137404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.137229 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.204069 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.203975 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvt74\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-kube-api-access-cvt74\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.204227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.204116 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.227725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.227695 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:06.241527 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.241494 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:06.739038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.738919 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt"] Nov 01 00:50:06.800894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.800248 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq"] Nov 01 00:50:06.801719 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:50:06.801440 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad1844c_94ac_45cf_b5db_13fc278f2f4a.slice/crio-4369cafa3da3fa29dcff59da67b45002dfe1369bf27504100aee5017cc1a2532 WatchSource:0}: Error finding container 4369cafa3da3fa29dcff59da67b45002dfe1369bf27504100aee5017cc1a2532: Status 404 returned error can't find the container with id 4369cafa3da3fa29dcff59da67b45002dfe1369bf27504100aee5017cc1a2532 Nov 01 00:50:06.802964 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:50:06.802942 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9454fc_6c85_497d_82c8_2adbbf3c56b7.slice/crio-cea208bc49c3d21cbbbbbc47f5546fb8d6dcb6535f7400c9ef9bdd5f51158593 WatchSource:0}: Error finding container cea208bc49c3d21cbbbbbc47f5546fb8d6dcb6535f7400c9ef9bdd5f51158593: Status 404 returned error can't find the container with id cea208bc49c3d21cbbbbbc47f5546fb8d6dcb6535f7400c9ef9bdd5f51158593 Nov 01 00:50:06.803977 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.803936 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:06.804085 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.804009 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:06.804085 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.804044 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:06.805674 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.805636 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:06.805755 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.805716 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:06.805795 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.805758 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:06.928119 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.928092 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 00:50:06.928504 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.928139 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 00:50:06.986252 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.986213 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:50:06.986678 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:06.986650 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 00:50:07.127711 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.127651 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw"] Nov 01 00:50:07.133652 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.133626 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.228844 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.228815 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw"] Nov 01 00:50:07.303317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.287567 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.303632 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.303607 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.304191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.303727 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855a861-3590-4e06-85d7-0bf38f8b463b-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.304191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.303792 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.304191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.303842 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.304191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.303989 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.304191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.304044 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.304191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.304136 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.304191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.304164 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mln67\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-kube-api-access-mln67\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.406504 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.406601 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.406631 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mln67\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-kube-api-access-mln67\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.406700 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.406844 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.406891 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855a861-3590-4e06-85d7-0bf38f8b463b-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.406942 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.406988 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.407126 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407954 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.407616 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.407954 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.407847 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.408378 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.408147 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.408516 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.408417 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.408573 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.408516 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855a861-3590-4e06-85d7-0bf38f8b463b-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.413902 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.413874 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.415600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.415575 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.492110 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.492052 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.492110 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.492061 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mln67\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-kube-api-access-mln67\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.526816 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.526787 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" event={"ID":"2a9454fc-6c85-497d-82c8-2adbbf3c56b7","Type":"ContainerStarted","Data":"1f5b4724aaa024d165e71ce21c70dbf46561261c0e6cac48a780f391d959fd2e"} Nov 01 00:50:07.527052 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.527030 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" event={"ID":"2a9454fc-6c85-497d-82c8-2adbbf3c56b7","Type":"ContainerStarted","Data":"cea208bc49c3d21cbbbbbc47f5546fb8d6dcb6535f7400c9ef9bdd5f51158593"} Nov 01 00:50:07.529629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.529603 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" event={"ID":"dad1844c-94ac-45cf-b5db-13fc278f2f4a","Type":"ContainerStarted","Data":"c59d3e2f365124168809caa676f7a0c4614a71978fbb5fe392b778e4776c80df"} Nov 01 00:50:07.529745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.529645 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" event={"ID":"dad1844c-94ac-45cf-b5db-13fc278f2f4a","Type":"ContainerStarted","Data":"4369cafa3da3fa29dcff59da67b45002dfe1369bf27504100aee5017cc1a2532"} Nov 01 00:50:07.787699 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.787620 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:07.800508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.800446 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podStartSLOduration=2.800429143 podStartE2EDuration="2.800429143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:07.79679926 +0000 UTC m=+1552.672020278" watchObservedRunningTime="2026-11-01 00:50:07.800429143 +0000 UTC m=+1552.675650157" Nov 01 00:50:07.801642 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.801601 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podStartSLOduration=2.801589087 podStartE2EDuration="2.801589087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:07.674133225 +0000 UTC m=+1552.549354264" watchObservedRunningTime="2026-11-01 00:50:07.801589087 +0000 UTC m=+1552.676810101" Nov 01 00:50:07.933446 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.928477 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 00:50:07.933446 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.928535 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 00:50:07.999311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.992568 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:50:07.999311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:07.992625 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 00:50:08.251533 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.251434 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 00:50:08.261315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.255368 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 00:50:08.372565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.372527 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 00:50:08.374419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.374382 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 00:50:08.386320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.262364 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:08.386320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.374685 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:08.444391 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.444350 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw"] Nov 01 00:50:08.533732 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.533673 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:08.534086 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.534060 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:08.534306 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.534245 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:08.708356 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.699320 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" event={"ID":"4855a861-3590-4e06-85d7-0bf38f8b463b","Type":"ContainerStarted","Data":"b77391aa96fc28be03c2a5b6815a042a35d04fe396f01514804de9dd9356d6af"} Nov 01 00:50:08.947914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.929695 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 00:50:08.947914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.929762 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 00:50:08.968977 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:08.968929 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 00:50:09.006317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.004829 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.029492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.029447 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.105421 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.105365 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.105705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.105679 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:50:09.105869 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.105839 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 00:50:09.117312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.109152 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 00:50:09.232372 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.231731 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 00:50:09.232372 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.231815 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 00:50:09.257403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.243962 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 00:50:09.257403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.244037 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 00:50:09.257403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.245031 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:50:09.324314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.302199 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.354313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.353278 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:50:09.651448 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.651388 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.676957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.676892 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.785027 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.784985 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.814377 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.814339 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" event={"ID":"4855a861-3590-4e06-85d7-0bf38f8b463b","Type":"ContainerStarted","Data":"5c2fa5e65c27e3eea5d52269e23da67c6597c4d4f9801349b8334f8f97a84d9d"} Nov 01 00:50:09.878911 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.849819 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.879207 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.855376 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:09.879475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.855412 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 00:50:09.929870 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.929770 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 00:50:09.930195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:09.930146 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 00:50:10.005312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.005083 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:10.071830 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.065315 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:10.071830 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.071526 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:50:10.071830 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.071600 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:50:10.097419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.074176 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:10.129448 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.129400 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:50:10.172315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.134841 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:50:10.278369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.153390 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.278635 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.278602 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 00:50:10.308178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.203333 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podStartSLOduration=3.203313752 podStartE2EDuration="3.203313752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:10.19591926 +0000 UTC m=+1555.071140281" watchObservedRunningTime="2026-11-01 00:50:10.203313752 +0000 UTC m=+1555.078534770" Nov 01 00:50:10.372314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.272981 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.154806 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.271833 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.385586 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.334011 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.385694 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.385733 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.272938 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.386432 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.315994 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:10.414408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.386699 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:50:10.434344 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.433577 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:50:10.439318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.438748 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:50:10.670315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.656967 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:50:10.670315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.658421 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:50:10.806312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.803425 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:10.919403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.893775 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:50:10.960625 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.960098 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:50:10.960625 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.938226 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.960625 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.960466 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 00:50:10.986171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.982808 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.986171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.982867 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:10.986171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.986030 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.986171 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:10.986075 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 00:50:11.080311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.072967 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:11.355584 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.355547 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 00:50:11.355851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.355816 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 00:50:11.364795 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.356031 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:11.387991 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.387948 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 00:50:11.409312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.389523 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:50:11.412373 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.412342 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 00:50:11.475923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.475881 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 00:50:11.476178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.449768 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:11.505384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.503968 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded" Nov 01 00:50:11.587324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.585547 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:11.675388 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.663593 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:50:11.838364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.761689 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Nov 01 00:50:11.871486 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.852592 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:11.871486 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.852652 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:11.872746 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.872710 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:11.916347 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.906617 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:11.966433 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:11.966332 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:50:12.076312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.072995 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:12.118540 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.118498 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 00:50:12.238382 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.237738 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 00:50:12.238382 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.237799 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 00:50:12.244790 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.244673 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 00:50:12.244790 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.244730 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 00:50:12.336309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.334410 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:12.385735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.365551 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:12.385735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.367020 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:50:12.421365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.410406 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:12.421365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.412088 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:12.453324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.450444 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:12.489309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.489158 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:12.489762 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.489741 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:50:12.569545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.558827 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:12.569545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.559579 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:12.569545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.564538 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 00:50:12.624532 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.624484 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 00:50:12.657269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.657233 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:50:12.670427 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:12.670381 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:50:13.091795 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.091759 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:13.121312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.094519 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:13.186312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.185760 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:13.202419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.187040 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:50:13.202419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.188961 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:13.202419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.189006 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:13.243349 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.213864 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 00:50:13.276314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.251199 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:13.286851 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.272262 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 00:50:13.287307 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.287132 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 00:50:13.340062 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.340009 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:13.421765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.421642 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:13.536827 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.506874 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:50:13.538792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.526795 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:13.539136 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.539106 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 00:50:13.539313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.539268 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 00:50:13.543522 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.543497 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:50:13.543617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.543556 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:50:13.543617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.543603 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:50:13.568525 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.543635 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:50:13.568525 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.543667 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:50:13.568525 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.543699 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:50:13.568525 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.543727 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:50:13.568525 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.553130 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:13.739387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.724400 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:13.922312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.805081 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:13.922312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.805137 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:13.922312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.845610 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:50:13.933319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:13.923775 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:50:14.001973 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.001877 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:50:14.030617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.030577 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:50:14.068301 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.068248 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:50:14.136605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.136576 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:50:14.241064 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.241009 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 00:50:14.241500 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.241469 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 00:50:14.241975 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.241953 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 00:50:14.242122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.242101 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 00:50:14.788843 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.788793 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:14.789047 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:14.788864 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:15.143358 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:15.142302 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g"] Nov 01 00:50:15.164428 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:15.164341 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cbd772-fqf7g"] Nov 01 00:50:15.252317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:15.244703 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 00:50:15.252619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:15.252585 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 00:50:15.309317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:15.267651 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 00:50:15.309317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:15.267711 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 00:50:15.434514 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:15.418140 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:15.668323 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:15.642046 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab75768e-995a-4bc2-877f-3016a9ba66ec" path="/var/lib/kubelet/pods/ab75768e-995a-4bc2-877f-3016a9ba66ec/volumes" Nov 01 00:50:16.021322 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.019964 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:16.021322 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.020046 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:16.186396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.186352 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:16.467442 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.467400 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:16.564324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.560938 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:16.585632 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.585581 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:16.756378 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.710378 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 01 00:50:16.756378 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.754627 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:50:16.756378 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.756130 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:50:16.780336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.777125 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:50:16.897175 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.893433 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:16.897175 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:16.893499 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:17.188485 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.188427 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.190092 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.190070 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:50:17.196918 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.196886 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.242194 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.242150 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.377314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.374597 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.401762 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.401724 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:17.406415 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.406370 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.478556 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.478425 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.479780 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.479742 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:17.484524 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.484500 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:50:17.488556 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.487426 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:17.531340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.525416 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:17.531340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.525489 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.533564 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.531787 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.649328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.587434 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:50:17.649328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.587914 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:50:17.649328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.616445 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.649328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.616840 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:50:17.763369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.761083 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:50:17.798159 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.798116 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:50:17.799155 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.798538 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:50:17.799982 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.798569 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:50:17.800738 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.798854 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:17.804127 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.804098 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:17.810705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.799719 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:50:17.810978 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.810956 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:50:17.811154 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.811140 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:50:17.884389 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:17.876430 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.136800 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.136760 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.364605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.345869 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:50:18.364605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.348742 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 00:50:18.364605 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.349098 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:50:18.384380 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.382739 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.384380 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.384051 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:50:18.418979 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.418876 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 00:50:18.472471 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.472432 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:18.472749 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.472719 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.482439 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.482395 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:18.482980 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.482570 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.509318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.483877 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:50:18.592443 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.592395 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:18.592726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.592695 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.601253 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.601220 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:50:18.626311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.622524 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.868368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.811832 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.868368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.812194 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.868368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.812518 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.868368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.815561 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:50:18.868368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.815903 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:50:18.880467 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.880428 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:18.899414 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.899363 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:18.905133 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.905103 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:50:18.949047 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.930774 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:50:18.949047 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.931390 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:50:18.993573 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:18.987379 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:19.049166 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.049131 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:19.049907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.049887 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:19.098770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.090627 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:50:19.144915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.124349 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:19.255159 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.255122 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:19.268999 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.268960 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:50:19.367014 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.366968 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:19.394058 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.383764 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:19.394876 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.383841 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:19.513101 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.498585 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 00:50:19.601689 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.601648 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:50:19.624417 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.624369 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:19.828108 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.828063 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.092005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.852793 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.092498 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.852831 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 01 00:50:20.092773 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.852857 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.093416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.852888 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.094030 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.854470 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:50:20.094183 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.094157 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 00:50:20.094465 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.964861 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:50:20.094675 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.094662 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:50:20.094809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.094799 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:50:20.094921 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.094912 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:50:20.103203 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:19.958064 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:20.104385 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:50:20.104355 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:50:20.106723 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.106707 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:50:20.132385 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.132349 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:50:20.160914 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.157571 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:50:20.185383 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.178535 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:50:20.186534 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.186508 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:50:20.191979 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.191959 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:50:20.193230 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.193216 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:50:20.967993 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.967962 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:20.968791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.968509 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:20.969766 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:20.969750 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:50:22.488492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:22.488448 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:27.584211 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:27.584175 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:29.181770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:29.181737 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:32.453498 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:32.453453 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:36.660764 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.660734 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9"] Nov 01 00:50:36.664885 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.664866 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.716848 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.716813 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr"] Nov 01 00:50:36.721309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.721272 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.768515 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.768489 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9"] Nov 01 00:50:36.828136 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.828097 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr"] Nov 01 00:50:36.850515 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.850479 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.850515 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.850519 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.850735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.850541 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.850775 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.850741 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.850880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.850864 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.850912 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.850890 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.850946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.850926 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdxr\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-kube-api-access-bjdxr\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.850992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.850983 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.851089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851067 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrb5\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-kube-api-access-9wrb5\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.851145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851125 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.851263 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851241 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.851343 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851278 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.851380 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851348 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.851380 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851366 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.851447 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851421 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc53d6-54da-4b31-8a11-c59e6216b215-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.851492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851450 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.851546 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851495 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.851625 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.851610 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.953470 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953381 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.953628 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953484 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.953628 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953515 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.953628 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953564 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdxr\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-kube-api-access-bjdxr\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.953628 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953618 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.953858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953653 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrb5\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-kube-api-access-9wrb5\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.953858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953700 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.953858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953774 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.953858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953799 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.953858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953831 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.953858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953855 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.954156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953887 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.954156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953903 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc53d6-54da-4b31-8a11-c59e6216b215-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.954156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953957 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.954156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953963 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.954156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.953984 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.954156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.954065 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.954156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.954129 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.954156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.954146 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.954595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.954165 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.954595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.954200 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.954595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.954371 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.954595 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.954432 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.954984 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.954918 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc53d6-54da-4b31-8a11-c59e6216b215-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.954984 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.955000 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.955214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.955054 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.955366 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.955342 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.955496 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.955477 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.955975 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.955948 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:36.956743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.956593 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.957295 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.957263 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:36.957814 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:36.957797 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:37.019668 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.019635 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:37.019868 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.019850 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrb5\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-kube-api-access-9wrb5\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:37.019946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.019851 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdxr\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-kube-api-access-bjdxr\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:37.020015 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.019997 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:37.036033 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.036009 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:37.279748 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.279662 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:37.363054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.363003 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:37.483563 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.483517 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr"] Nov 01 00:50:37.483710 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:50:37.483654 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adc53d6_54da_4b31_8a11_c59e6216b215.slice/crio-9e711aa21b83af0412ad574f052371be1311e499ea9e0eec7253cb1aa661d468 WatchSource:0}: Error finding container 9e711aa21b83af0412ad574f052371be1311e499ea9e0eec7253cb1aa661d468: Status 404 returned error can't find the container with id 9e711aa21b83af0412ad574f052371be1311e499ea9e0eec7253cb1aa661d468 Nov 01 00:50:37.487219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.487182 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.487352 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.487255 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.487352 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.487313 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.653816 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.653784 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2"] Nov 01 00:50:37.659156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.659111 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.664412 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:50:37.664386 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf10d2f9_c3ea_43e1_bb2e_9345697fbccc.slice/crio-bebf3f777eb78d1ce2f4e8740c550db06decaff65b312736c04360654460985d WatchSource:0}: Error finding container bebf3f777eb78d1ce2f4e8740c550db06decaff65b312736c04360654460985d: Status 404 returned error can't find the container with id bebf3f777eb78d1ce2f4e8740c550db06decaff65b312736c04360654460985d Nov 01 00:50:37.665873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.665844 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9"] Nov 01 00:50:37.667450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667423 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.667543 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667456 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.667543 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667477 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.667543 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667533 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.667710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667574 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8xv\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-kube-api-access-jp8xv\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.667710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667578 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.667710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667638 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.667863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667716 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.667863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667776 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.667863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667849 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.668014 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.667958 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.668620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.668123 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.721509 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.721470 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2"] Nov 01 00:50:37.740697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.739803 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" event={"ID":"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc","Type":"ContainerStarted","Data":"bebf3f777eb78d1ce2f4e8740c550db06decaff65b312736c04360654460985d"} Nov 01 00:50:37.749406 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.749368 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" event={"ID":"0adc53d6-54da-4b31-8a11-c59e6216b215","Type":"ContainerStarted","Data":"5038517ecb36ade046090024bc922ba373a71f292552ffcc694112908b14c55e"} Nov 01 00:50:37.749532 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.749441 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" event={"ID":"0adc53d6-54da-4b31-8a11-c59e6216b215","Type":"ContainerStarted","Data":"9e711aa21b83af0412ad574f052371be1311e499ea9e0eec7253cb1aa661d468"} Nov 01 00:50:37.769955 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.769917 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.770143 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.770026 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.770143 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.770098 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.770143 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.770140 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.770338 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.770168 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8xv\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-kube-api-access-jp8xv\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.770338 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.770205 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.770338 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.770315 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.770505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.770403 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.770505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.770492 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.772324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.771238 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.772324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.771368 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.772324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.772086 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.772324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.772124 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.772324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.772264 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.779673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.779612 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.780015 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.779885 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.836353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.835725 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.836353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.835875 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8xv\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-kube-api-access-jp8xv\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:37.910036 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.909981 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podStartSLOduration=1.909965476 podStartE2EDuration="1.909965476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:37.908789537 +0000 UTC m=+1582.784010553" watchObservedRunningTime="2026-11-01 00:50:37.909965476 +0000 UTC m=+1582.785186490" Nov 01 00:50:37.981262 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:37.981216 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:38.037035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.037000 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:38.038496 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.038472 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 00:50:38.038618 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.038531 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 00:50:38.407552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.407520 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2"] Nov 01 00:50:38.411634 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:50:38.411584 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5a3620_fb09_4cb9_95e2_ddafe9e12a57.slice/crio-0cb4eb51984c1f17a34b6cc68e826bafd5ebf03c6a1f142f150c0de12bb8ba6a WatchSource:0}: Error finding container 0cb4eb51984c1f17a34b6cc68e826bafd5ebf03c6a1f142f150c0de12bb8ba6a: Status 404 returned error can't find the container with id 0cb4eb51984c1f17a34b6cc68e826bafd5ebf03c6a1f142f150c0de12bb8ba6a Nov 01 00:50:38.422720 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.422589 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:38.422869 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.422767 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:38.422869 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.422854 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:38.791888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.791855 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" event={"ID":"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57","Type":"ContainerStarted","Data":"eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a"} Nov 01 00:50:38.792545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.792522 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" event={"ID":"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57","Type":"ContainerStarted","Data":"0cb4eb51984c1f17a34b6cc68e826bafd5ebf03c6a1f142f150c0de12bb8ba6a"} Nov 01 00:50:38.800315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.799418 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" event={"ID":"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc","Type":"ContainerStarted","Data":"458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3"} Nov 01 00:50:38.934585 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.934521 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podStartSLOduration=1.934500743 podStartE2EDuration="1.934500743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:38.931631102 +0000 UTC m=+1583.806852119" watchObservedRunningTime="2026-11-01 00:50:38.934500743 +0000 UTC m=+1583.809721763" Nov 01 00:50:38.995364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:38.984398 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:39.037383 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:39.037345 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 00:50:39.037622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:39.037413 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 00:50:39.090954 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:39.090819 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podStartSLOduration=3.090790631 podStartE2EDuration="3.090790631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:39.066622059 +0000 UTC m=+1583.941843088" watchObservedRunningTime="2026-11-01 00:50:39.090790631 +0000 UTC m=+1583.966011647" Nov 01 00:50:39.293314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:39.290900 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 00:50:39.293314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:39.290964 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 00:50:39.308904 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:39.306753 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:39.986313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:39.986188 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 00:50:39.986313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:39.986250 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 00:50:40.036606 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:40.036479 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 00:50:40.036606 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:40.036537 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 00:50:40.301311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:40.286833 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 00:50:40.301311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:40.286907 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 00:50:40.982311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:40.981827 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 00:50:40.982311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:40.981909 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 00:50:41.036583 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.036549 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 00:50:41.036955 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.036600 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 00:50:41.288375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.286428 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 00:50:41.288375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.286486 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 00:50:41.325313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.323402 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl"] Nov 01 00:50:41.325313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.324041 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" containerID="cri-o://cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426" gracePeriod=30 Nov 01 00:50:41.435099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.383943 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94"] Nov 01 00:50:41.435099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.390334 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" containerID="cri-o://eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5" gracePeriod=30 Nov 01 00:50:41.441197 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.440886 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc"] Nov 01 00:50:41.441446 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.441268 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" containerID="cri-o://23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa" gracePeriod=30 Nov 01 00:50:41.444491 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.444465 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp"] Nov 01 00:50:41.444673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.444652 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" containerID="cri-o://5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd" gracePeriod=30 Nov 01 00:50:41.987311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.986703 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 00:50:41.987311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:41.986805 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 00:50:42.036541 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:42.036499 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 00:50:42.036743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:42.036594 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 00:50:42.283756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:42.283717 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 00:50:42.284027 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:42.283994 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 00:50:42.629585 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:42.629477 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:42.985371 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:42.984680 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 00:50:42.985371 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:42.984741 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 00:50:43.037164 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:43.037127 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 00:50:43.037629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:43.037201 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 00:50:43.307314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:43.298180 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 00:50:43.307314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:43.298240 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 00:50:44.004877 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:44.004190 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 00:50:44.005192 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:44.005157 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 00:50:44.037055 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:44.037022 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 00:50:44.037251 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:44.037078 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 00:50:44.280859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:44.280751 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 00:50:44.280859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:44.280822 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 00:50:44.981741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:44.981705 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 00:50:44.981961 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:44.981761 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 00:50:45.412753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:45.411964 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:45.415227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:45.413711 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:45.415988 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:45.415964 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 00:50:45.674087 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:45.673997 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:45.674779 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:45.674746 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:45.675726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:45.675705 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 00:50:45.982396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:45.982273 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 00:50:45.982396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:45.982379 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 00:50:46.681191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.681163 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:50:46.693022 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.692990 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:50:46.792309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.791606 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-data\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.792309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.791693 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-certs\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.792309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.791741 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-data\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.792309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.791961 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-data" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.792309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.791984 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-certs" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.792309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.792001 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-data" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.792309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.792056 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-credential-socket\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.792801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.792383 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-credential-socket" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.792801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.792437 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-podinfo\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.792801 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.792755 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-envoy\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.792965 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.792927 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-envoy\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.793020 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.792989 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-credential-socket\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.793077 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793019 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-token\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-token\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.793132 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793091 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-token\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-token\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.793185 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793172 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-certs\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793235 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-podinfo\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793272 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef75179e-d5ea-4254-a616-6f2f271728d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef75179e-d5ea-4254-a616-6f2f271728d2-istiod-ca-cert\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793420 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-kube-api-access-khkgr\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-kube-api-access-khkgr\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793493 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6313eca7-5cd0-472f-be31-2860c6b47e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6313eca7-5cd0-472f-be31-2860c6b47e65-istiod-ca-cert\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793524 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-socket\") pod \"6313eca7-5cd0-472f-be31-2860c6b47e65\" (UID: \"6313eca7-5cd0-472f-be31-2860c6b47e65\") " Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793581 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-kube-api-access-7b49w\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-kube-api-access-7b49w\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.793929 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-socket\") pod \"ef75179e-d5ea-4254-a616-6f2f271728d2\" (UID: \"ef75179e-d5ea-4254-a616-6f2f271728d2\") " Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.794055 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-certs" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.794443 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef75179e-d5ea-4254-a616-6f2f271728d2-istiod-ca-cert" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.794447 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6313eca7-5cd0-472f-be31-2860c6b47e65-istiod-ca-cert" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.794882 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-credential-socket" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795187 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-socket" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795379 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-socket" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795631 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795654 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef75179e-d5ea-4254-a616-6f2f271728d2-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795673 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6313eca7-5cd0-472f-be31-2860c6b47e65-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795688 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795704 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795717 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795730 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795742 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795754 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.801342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.795770 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.802843 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.802798 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-kube-api-access-khkgr" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "kube-api-access-khkgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:46.803097 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.802847 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-kube-api-access-7b49w" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "kube-api-access-7b49w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:46.803097 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.802867 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-token" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:46.803097 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.802969 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-token" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:46.803097 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.803005 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-podinfo" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:50:46.803097 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.803086 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-envoy" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.804680 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.804657 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-envoy" pod "ef75179e-d5ea-4254-a616-6f2f271728d2" (UID: "ef75179e-d5ea-4254-a616-6f2f271728d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.804977 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.804954 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-podinfo" pod "6313eca7-5cd0-472f-be31-2860c6b47e65" (UID: "6313eca7-5cd0-472f-be31-2860c6b47e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:50:46.835789 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.835738 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:50:46.859223 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.859199 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:50:46.897888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.897853 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.897888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.897884 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkgr\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-kube-api-access-khkgr\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.898076 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.897900 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b49w\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-kube-api-access-7b49w\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.898076 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.897913 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.898076 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.897927 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.898076 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.897938 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.898076 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.897952 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef75179e-d5ea-4254-a616-6f2f271728d2-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.898076 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.897964 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6313eca7-5cd0-472f-be31-2860c6b47e65-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:46.986234 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.986207 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:46.986641 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.986607 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:46.987361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.987331 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:50:46.998854 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.998832 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e22a6be0-4627-418a-aef8-38900f0445d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22a6be0-4627-418a-aef8-38900f0445d7-istio-podinfo\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:46.998992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.998864 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-kube-api-access-n4jwp\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-kube-api-access-n4jwp\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:46.998992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.998884 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-credential-socket\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:46.998992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.998917 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-certs\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:46.999175 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999043 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-certs\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:46.999175 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999128 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-data\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:46.999308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999175 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-credential-socket" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.999308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999188 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-credential-socket\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:46.999308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999189 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-certs" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.999308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999277 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-certs" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.999481 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999384 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-credential-socket" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.999481 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999391 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-envoy\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:46.999481 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999403 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-data" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.999481 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999425 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-socket\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:46.999665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999482 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istiod-ca-cert\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:46.999665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999539 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22a6be0-4627-418a-aef8-38900f0445d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22a6be0-4627-418a-aef8-38900f0445d7-istiod-ca-cert\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:46.999665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999597 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-socket" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:46.999847 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999825 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istiod-ca-cert" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:46.999847 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:46.999829 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22a6be0-4627-418a-aef8-38900f0445d7-istiod-ca-cert" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:47.000116 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.000092 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-data\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:47.000232 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.000146 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-token\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-token\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:47.000232 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.000210 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-podinfo\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:47.000447 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.000252 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-envoy\") pod \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\" (UID: \"0e7b1369-cee3-4dac-bd97-fabb32d2409c\") " Nov 01 00:50:47.000447 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.000413 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-data" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:47.000619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.000480 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-kube-api-access-pzwkp\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-kube-api-access-pzwkp\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:47.000619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.000529 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-istio-token\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-istio-token\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:47.000619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.000559 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-socket\") pod \"e22a6be0-4627-418a-aef8-38900f0445d7\" (UID: \"e22a6be0-4627-418a-aef8-38900f0445d7\") " Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001409 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001435 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001452 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001466 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001479 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001493 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001506 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001519 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22a6be0-4627-418a-aef8-38900f0445d7-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001533 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.001812 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-socket" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:47.002048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.002001 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-kube-api-access-n4jwp" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "kube-api-access-n4jwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:47.002729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.002053 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-envoy" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:47.002729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.002552 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e22a6be0-4627-418a-aef8-38900f0445d7-istio-podinfo" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:50:47.003053 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.003022 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-podinfo" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:50:47.003224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.003205 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-kube-api-access-pzwkp" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "kube-api-access-pzwkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:47.003874 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.003845 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-envoy" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:50:47.004030 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.004011 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-token" pod "0e7b1369-cee3-4dac-bd97-fabb32d2409c" (UID: "0e7b1369-cee3-4dac-bd97-fabb32d2409c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:47.004871 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.004850 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-istio-token" pod "e22a6be0-4627-418a-aef8-38900f0445d7" (UID: "e22a6be0-4627-418a-aef8-38900f0445d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:47.037051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.037007 6036 generic.go:356] "Generic (PLEG): container finished" podID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerID="cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426" exitCode=0 Nov 01 00:50:47.037244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.037120 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" Nov 01 00:50:47.037244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.037151 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" event={"ID":"6313eca7-5cd0-472f-be31-2860c6b47e65","Type":"ContainerDied","Data":"cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426"} Nov 01 00:50:47.037244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.037206 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl" event={"ID":"6313eca7-5cd0-472f-be31-2860c6b47e65","Type":"ContainerDied","Data":"307bb8392b815749e01b2bf143ef78b305d79a1406c6829ce5395ed72080b175"} Nov 01 00:50:47.037244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.037235 6036 scope.go:122] "RemoveContainer" containerID="cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426" Nov 01 00:50:47.045530 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.045495 6036 generic.go:356] "Generic (PLEG): container finished" podID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerID="eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5" exitCode=0 Nov 01 00:50:47.045714 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.045689 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" event={"ID":"ef75179e-d5ea-4254-a616-6f2f271728d2","Type":"ContainerDied","Data":"eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5"} Nov 01 00:50:47.045819 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.045743 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" event={"ID":"ef75179e-d5ea-4254-a616-6f2f271728d2","Type":"ContainerDied","Data":"62ceb80e332b5ab3ff60d525ee28e1ac124a0b8f631d79d3f4d4396eecb48c00"} Nov 01 00:50:47.045819 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.045810 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94" Nov 01 00:50:47.048738 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.048715 6036 generic.go:356] "Generic (PLEG): container finished" podID="e22a6be0-4627-418a-aef8-38900f0445d7" containerID="5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd" exitCode=0 Nov 01 00:50:47.048864 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.048849 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" event={"ID":"e22a6be0-4627-418a-aef8-38900f0445d7","Type":"ContainerDied","Data":"5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd"} Nov 01 00:50:47.048928 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.048886 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" event={"ID":"e22a6be0-4627-418a-aef8-38900f0445d7","Type":"ContainerDied","Data":"1bf4c569cdcbfa7a9b580006e6b9e6cc25cf1588b08c285cf6d144a5fa86e5aa"} Nov 01 00:50:47.048982 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.048949 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp" Nov 01 00:50:47.055695 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.055425 6036 scope.go:122] "RemoveContainer" containerID="cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426" Nov 01 00:50:47.055876 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:50:47.055851 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426\": container with ID starting with cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426 not found: ID does not exist" containerID="cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426" Nov 01 00:50:47.055940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.055890 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426"} err="failed to get container status \"cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426\": rpc error: code = NotFound desc = could not find container \"cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426\": container with ID starting with cefffd954e015041bd2acdbe4e905f4afc0f61142adc86fc7724aa9d6db44426 not found: ID does not exist" Nov 01 00:50:47.055940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.055914 6036 scope.go:122] "RemoveContainer" containerID="eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5" Nov 01 00:50:47.057353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.057211 6036 generic.go:356] "Generic (PLEG): container finished" podID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerID="23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa" exitCode=0 Nov 01 00:50:47.057353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.057267 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" event={"ID":"0e7b1369-cee3-4dac-bd97-fabb32d2409c","Type":"ContainerDied","Data":"23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa"} Nov 01 00:50:47.057492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.057352 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" event={"ID":"0e7b1369-cee3-4dac-bd97-fabb32d2409c","Type":"ContainerDied","Data":"4939a83f905c3c4e09dce6c47cf7b4ff00db6804d257f354caa8fb767a5503c4"} Nov 01 00:50:47.057492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.057276 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc" Nov 01 00:50:47.068154 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.068131 6036 scope.go:122] "RemoveContainer" containerID="eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5" Nov 01 00:50:47.068629 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:50:47.068599 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5\": container with ID starting with eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5 not found: ID does not exist" containerID="eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5" Nov 01 00:50:47.068702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.068639 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5"} err="failed to get container status \"eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5\": rpc error: code = NotFound desc = could not find container \"eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5\": container with ID starting with eaee84e641e5d7f2974be90b90edb63d18bf4b2a386ecc2825a80b7696cfa9c5 not found: ID does not exist" Nov 01 00:50:47.068702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.068662 6036 scope.go:122] "RemoveContainer" containerID="5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd" Nov 01 00:50:47.079582 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.079553 6036 scope.go:122] "RemoveContainer" containerID="5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd" Nov 01 00:50:47.079907 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:50:47.079884 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd\": container with ID starting with 5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd not found: ID does not exist" containerID="5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd" Nov 01 00:50:47.080005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.079915 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd"} err="failed to get container status \"5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd\": rpc error: code = NotFound desc = could not find container \"5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd\": container with ID starting with 5de08eee26a87042d03948fd302f8c24dc13e15be92e21df036308266dc148bd not found: ID does not exist" Nov 01 00:50:47.080005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.079939 6036 scope.go:122] "RemoveContainer" containerID="23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa" Nov 01 00:50:47.093337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.093178 6036 scope.go:122] "RemoveContainer" containerID="23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa" Nov 01 00:50:47.093638 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:50:47.093616 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa\": container with ID starting with 23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa not found: ID does not exist" containerID="23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa" Nov 01 00:50:47.093709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.093647 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa"} err="failed to get container status \"23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa\": rpc error: code = NotFound desc = could not find container \"23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa\": container with ID starting with 23b4955c1be005c381f7653941b56047197de4e21e45c54928a67de79c10afaa not found: ID does not exist" Nov 01 00:50:47.102880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102808 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.102880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102840 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.102880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102859 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7b1369-cee3-4dac-bd97-fabb32d2409c-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.102880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102875 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzwkp\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-kube-api-access-pzwkp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.103103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102893 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22a6be0-4627-418a-aef8-38900f0445d7-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.103103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102907 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.103103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102924 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22a6be0-4627-418a-aef8-38900f0445d7-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.103103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102939 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jwp\" (UniqueName: \"kubernetes.io/projected/0e7b1369-cee3-4dac-bd97-fabb32d2409c-kube-api-access-n4jwp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.103103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.102952 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22a6be0-4627-418a-aef8-38900f0445d7-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.399230 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.399186 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl"] Nov 01 00:50:47.402495 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.402475 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzxfl"] Nov 01 00:50:47.511377 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.511339 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp"] Nov 01 00:50:47.514967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.514937 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6ljpp"] Nov 01 00:50:47.753131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.752986 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:47.759792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.759758 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94"] Nov 01 00:50:47.764116 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.764073 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-v8p94"] Nov 01 00:50:47.872804 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.872765 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc"] Nov 01 00:50:47.879148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:47.879116 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-tn2pc"] Nov 01 00:50:49.253190 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:49.253156 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" path="/var/lib/kubelet/pods/0e7b1369-cee3-4dac-bd97-fabb32d2409c/volumes" Nov 01 00:50:49.253711 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:49.253696 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" path="/var/lib/kubelet/pods/6313eca7-5cd0-472f-be31-2860c6b47e65/volumes" Nov 01 00:50:49.254126 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:49.254114 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" path="/var/lib/kubelet/pods/e22a6be0-4627-418a-aef8-38900f0445d7/volumes" Nov 01 00:50:49.254882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:49.254867 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" path="/var/lib/kubelet/pods/ef75179e-d5ea-4254-a616-6f2f271728d2/volumes" Nov 01 00:50:49.507866 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:50:49.507784 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:50:52.120369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.120334 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch"] Nov 01 00:50:52.120952 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.120938 6036 cpu_manager.go:414] "containerMap: removing container" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" Nov 01 00:50:52.121000 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.120953 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" Nov 01 00:50:52.121000 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.120968 6036 cpu_manager.go:414] "containerMap: removing container" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" Nov 01 00:50:52.121000 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.120973 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" Nov 01 00:50:52.121000 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.120996 6036 cpu_manager.go:414] "containerMap: removing container" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" Nov 01 00:50:52.121117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.121002 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" Nov 01 00:50:52.121117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.121022 6036 cpu_manager.go:414] "containerMap: removing container" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" Nov 01 00:50:52.121117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.121027 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" Nov 01 00:50:52.121203 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.121123 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6313eca7-5cd0-472f-be31-2860c6b47e65" containerName="istio-proxy" Nov 01 00:50:52.121203 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.121134 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef75179e-d5ea-4254-a616-6f2f271728d2" containerName="istio-proxy" Nov 01 00:50:52.121203 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.121143 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22a6be0-4627-418a-aef8-38900f0445d7" containerName="istio-proxy" Nov 01 00:50:52.121203 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.121151 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7b1369-cee3-4dac-bd97-fabb32d2409c" containerName="istio-proxy" Nov 01 00:50:52.126394 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.126370 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.182846 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.182808 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.183061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.182925 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7cfb12-70c6-4469-9c8a-5997786d2125-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.183061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.183003 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzxq\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-kube-api-access-rmzxq\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.183061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.183048 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.183231 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.183159 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.183231 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.183182 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.183384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.183255 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.183384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.183319 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.183384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.183374 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.206341 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.187266 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch"] Nov 01 00:50:52.285418 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285379 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.285618 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285467 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.285618 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285534 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.285749 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285653 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7cfb12-70c6-4469-9c8a-5997786d2125-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.285749 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285733 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.285749 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285742 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzxq\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-kube-api-access-rmzxq\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.285922 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285798 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.285922 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285827 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.285922 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285915 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.286090 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285928 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.286090 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.285933 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.286199 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.286148 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.286263 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.286191 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.286438 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.286415 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7cfb12-70c6-4469-9c8a-5997786d2125-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.288368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.288345 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.288584 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.288563 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.348626 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.348593 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.348778 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.348662 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzxq\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-kube-api-access-rmzxq\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.440353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.440243 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:52.690013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.689982 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:50:52.885071 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:50:52.885025 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7cfb12_70c6_4469_9c8a_5997786d2125.slice/crio-7b31a7c3bb9d123e76f2a8d3e326626e0121fc349330d9bab4369395466f4e69 WatchSource:0}: Error finding container 7b31a7c3bb9d123e76f2a8d3e326626e0121fc349330d9bab4369395466f4e69: Status 404 returned error can't find the container with id 7b31a7c3bb9d123e76f2a8d3e326626e0121fc349330d9bab4369395466f4e69 Nov 01 00:50:52.885355 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.885332 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch"] Nov 01 00:50:52.887870 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.887838 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:52.887943 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.887904 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:52.887943 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:52.887932 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:53.107541 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:53.107504 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" event={"ID":"9d7cfb12-70c6-4469-9c8a-5997786d2125","Type":"ContainerStarted","Data":"6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792"} Nov 01 00:50:53.107726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:53.107566 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" event={"ID":"9d7cfb12-70c6-4469-9c8a-5997786d2125","Type":"ContainerStarted","Data":"7b31a7c3bb9d123e76f2a8d3e326626e0121fc349330d9bab4369395466f4e69"} Nov 01 00:50:53.231453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:53.231330 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podStartSLOduration=1.23131266 podStartE2EDuration="1.23131266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:53.23123667 +0000 UTC m=+1598.106457684" watchObservedRunningTime="2026-11-01 00:50:53.23131266 +0000 UTC m=+1598.106533673" Nov 01 00:50:53.441379 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:53.441345 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:50:53.443213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:53.443188 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:50:53.443339 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:53.443233 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:50:54.473340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:54.453515 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:50:54.473340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:54.453577 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:50:55.184513 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.184455 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.248127 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.248062 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:50:55.393399 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.377430 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.494465 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.493608 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.527311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.523940 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.623519 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.623470 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:50:55.659327 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.647809 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.659327 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.648515 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.688419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.676159 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:50:55.746741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.746082 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:50:55.746741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.746150 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:50:55.825260 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.823650 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.825628 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.825596 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.912398 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:55.911882 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.190340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.188920 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:50:56.190340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.189375 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:50:56.426243 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.426191 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.452516 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.452418 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:50:56.504353 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.475922 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.505067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.505022 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:50:56.534684 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.534621 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:50:56.535075 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.535008 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:50:56.728369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.727113 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.728369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.728310 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.732321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.729462 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:50:56.732321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.731176 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 01 00:50:56.732321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.731236 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 00:50:56.755531 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.755467 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:50:56.813034 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.755802 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:50:56.837316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.755905 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:50:56.845323 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.841430 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:50:56.875466 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.875429 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:50:56.875765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.875746 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:50:56.875924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.875910 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:50:56.879313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.876052 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:50:56.942312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.826432 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.962049 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.826503 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.992534 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.992442 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:50:56.992825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.992806 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:50:56.997314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:56.929574 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:57.014323 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.010580 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:50:57.190474 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.190427 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:50:57.317660 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.317602 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.450779 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.437990 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:50:57.529315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.488113 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.529315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.499278 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:50:57.529315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.499357 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:50:57.752373 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.749991 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:50:57.806313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.782223 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.859722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.784463 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.863309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.784521 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.892308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.814431 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:50:57.950112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.817002 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.978149 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:57.963776 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.022395 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.012010 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.078313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.047392 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:50:58.166207 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.166158 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.201316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.197366 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:50:58.411389 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.405703 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 00:50:58.439332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.428438 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:58.439332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.428523 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.455329 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.454448 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.455329 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.455081 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:50:58.455329 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.455127 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:50:58.505345 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.500952 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 00:50:58.751505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.751371 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:50:58.862313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.859826 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.862313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.860234 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.038376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.991561 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 00:50:59.038376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.991650 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:50:59.038376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.992375 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 00:50:59.038376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.993013 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.038376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:58.993253 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.038376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.030642 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:59.102311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.094237 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.177311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.162026 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:59.177311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.162093 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.230763 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.206385 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.529782 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.529673 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.530418 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.530387 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:50:59.542796 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.542750 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:51:00.155491 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.155440 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:51:00.156099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.811449 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 01 00:51:00.156224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.953199 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:00.168312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.767567 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.298316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.028778 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:00.311318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.028819 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:51:00.311811 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.036602 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.334317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.039024 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.373311 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:51:00.341275 6036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.097s" Nov 01 00:51:00.373311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.126209 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.373311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.953241 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 00:51:00.373311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:50:59.631988 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:00.401314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.258130 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.401314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.395864 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.404118 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.040397 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.449368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.445963 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:00.481053 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.481006 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.481356 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.455708 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:00.481646 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.447656 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:51:00.481743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.481688 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:51:00.547312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.539266 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.547312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.539786 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.547312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.543457 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.716126 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:00.716034 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:01.030505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.030388 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.157443 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.157388 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.314684 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.314641 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 00:51:01.325007 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.324952 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:51:01.337067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.336182 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:51:01.337067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.336553 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:01.337067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.336775 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:51:01.375320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.365007 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.399311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.390668 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.406713 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.406671 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.407025 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.406606 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.538312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.514796 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:51:01.538312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.514858 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:51:01.538312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.526807 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:01.538312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.526884 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.538312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.527307 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.549689 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.543790 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.563311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.552211 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.757751 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.755274 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.760395 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.760361 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:01.760619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:01.760592 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.051784 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.051112 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 01 00:51:02.188359 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.165532 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.335311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.332307 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 00:51:02.336194 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.335574 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.349861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.336405 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.349861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.341015 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.349861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.341661 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:02.366761 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.366378 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.412360 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.407561 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:51:02.412360 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.408345 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.412360 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.409458 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 00:51:02.546309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.442992 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:51:02.546309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.443064 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:51:02.546309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.445430 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:51:02.556981 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.556318 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.571558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.571531 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:51:02.586957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.585471 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.790828 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.790800 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:51:02.840657 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.839687 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:51:02.891511 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.891470 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:51:02.901000 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.900974 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:51:02.922563 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.921938 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:51:02.928545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.928524 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:51:02.929561 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.929544 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:51:02.936227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.936102 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:51:02.936227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.936159 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:51:02.944123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:02.943899 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:51:03.441003 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:03.440960 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:51:03.441398 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:03.441022 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:51:04.441141 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:04.441094 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:51:04.441702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:04.441152 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:51:04.571440 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:04.567588 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:05.440899 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:05.440865 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 00:51:05.441080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:05.440920 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 00:51:07.324230 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.324193 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:07.440694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.440657 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:51:07.481513 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.481463 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:07.481701 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.481533 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:07.547919 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.547891 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:51:07.557664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.557610 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:07.573710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.573680 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br"] Nov 01 00:51:07.578371 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.578325 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.656021 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.655991 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br"] Nov 01 00:51:07.691315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691260 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.691461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691386 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.691461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691420 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.691556 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691501 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.691556 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691550 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.691628 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691593 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.691741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691720 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.691810 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691785 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.691913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.691894 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxpqj\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-kube-api-access-rxpqj\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.766738 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.766702 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:07.767002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.766983 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:51:07.793601 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.793562 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxpqj\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-kube-api-access-rxpqj\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.793773 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.793751 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.793825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.793808 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.793875 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.793846 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.793920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.793895 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.793983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.793971 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.794077 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.794058 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.794233 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.794206 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.794467 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.794422 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.794588 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.794495 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.794674 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.794609 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.794753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.794690 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.794753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.794727 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.794988 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.794963 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.797035 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.797006 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.797583 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.797558 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.857880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.857803 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxpqj\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-kube-api-access-rxpqj\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.858037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.857991 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:07.893461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:07.893430 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:08.340164 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.340135 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br"] Nov 01 00:51:08.340511 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:51:08.340144 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21aa9c83_33c1_4471_9bfc_8d0eab7459ae.slice/crio-2c704aa09aac6f73a488cbc7400197b9141600af14823f1d6a539b3859a3fe58 WatchSource:0}: Error finding container 2c704aa09aac6f73a488cbc7400197b9141600af14823f1d6a539b3859a3fe58: Status 404 returned error can't find the container with id 2c704aa09aac6f73a488cbc7400197b9141600af14823f1d6a539b3859a3fe58 Nov 01 00:51:08.343144 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.343106 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:08.343247 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.343179 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:08.343247 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.343223 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:08.373649 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.373618 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:08.506585 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.506553 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f"] Nov 01 00:51:08.510968 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.510943 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.570879 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.570845 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-64566096-openshift-default-dockercfg-g8w4m\"" Nov 01 00:51:08.574482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.574449 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f"] Nov 01 00:51:08.612529 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612439 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-envoy\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.612529 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612494 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-certs\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.612698 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612582 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istiod-ca-cert\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.612736 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612700 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-credential-socket\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.612736 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612724 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-data\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.612805 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612762 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-token\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.612845 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612818 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-podinfo\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.612881 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612852 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjfw\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-kube-api-access-6sjfw\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.612952 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.612937 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-socket\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.701261 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.701225 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:08.714791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.714758 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-envoy\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.714967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.714798 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-certs\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.714967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.714854 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istiod-ca-cert\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.714967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.714938 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-credential-socket\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.714967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.714959 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-data\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.715142 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.715086 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-token\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.715189 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.715143 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-podinfo\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.715189 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.715177 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjfw\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-kube-api-access-6sjfw\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.715359 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.715258 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-socket\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.715724 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.715459 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-credential-socket\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.715867 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.715753 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-certs\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.715867 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.715837 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-socket\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.715978 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.715908 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istiod-ca-cert\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.716122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.716094 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-data\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.717949 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.717929 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-envoy\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.718186 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.718167 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-podinfo\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.797173 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.796928 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjfw\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-kube-api-access-6sjfw\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.798076 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.798047 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-token\") pod \"e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.805443 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.805407 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" event={"ID":"21aa9c83-33c1-4471-9bfc-8d0eab7459ae","Type":"ContainerStarted","Data":"2fa20f32188956356216cda72bdc8c5996769bc6c86fa91954876551803c44b8"} Nov 01 00:51:08.805636 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.805535 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" event={"ID":"21aa9c83-33c1-4471-9bfc-8d0eab7459ae","Type":"ContainerStarted","Data":"2c704aa09aac6f73a488cbc7400197b9141600af14823f1d6a539b3859a3fe58"} Nov 01 00:51:08.831636 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.831346 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:08.894165 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:08.894075 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:09.056361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.038609 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:09.310039 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.309642 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podStartSLOduration=2.3096220499999998 podStartE2EDuration="2.30962205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:09.045855818 +0000 UTC m=+1613.921076837" watchObservedRunningTime="2026-11-01 00:51:09.30962205 +0000 UTC m=+1614.184843067" Nov 01 00:51:09.312440 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.312414 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f"] Nov 01 00:51:09.312864 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:51:09.312839 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8880c8e_a9f3_41b3_87c6_cf1b8b49d47d.slice/crio-4c0bf326d6ec8d3b0faea4c99a60c0a48eff5d2355aa3b6ae55aef1ff5d67d33 WatchSource:0}: Error finding container 4c0bf326d6ec8d3b0faea4c99a60c0a48eff5d2355aa3b6ae55aef1ff5d67d33: Status 404 returned error can't find the container with id 4c0bf326d6ec8d3b0faea4c99a60c0a48eff5d2355aa3b6ae55aef1ff5d67d33 Nov 01 00:51:09.315727 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.315685 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:09.315816 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.315779 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:09.315855 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.315822 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:09.506528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.506491 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:09.806312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.795203 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:09.822960 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.822926 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" event={"ID":"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d","Type":"ContainerStarted","Data":"e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676"} Nov 01 00:51:09.823106 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.822983 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" event={"ID":"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d","Type":"ContainerStarted","Data":"4c0bf326d6ec8d3b0faea4c99a60c0a48eff5d2355aa3b6ae55aef1ff5d67d33"} Nov 01 00:51:09.831761 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.831725 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:09.901865 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.901826 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 00:51:09.902158 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:09.901886 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 00:51:10.282721 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:10.282687 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:10.837528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:10.837494 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:10.894675 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:10.894640 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 00:51:10.894863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:10.894708 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 00:51:11.022676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:11.022623 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podStartSLOduration=3.022606422 podStartE2EDuration="3.022606422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:09.972955215 +0000 UTC m=+1614.848176231" watchObservedRunningTime="2026-11-01 00:51:11.022606422 +0000 UTC m=+1615.897827440" Nov 01 00:51:11.836888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:11.836860 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:11.838209 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:11.838181 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:51:11.894084 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:11.894055 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 00:51:11.894275 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:11.894103 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 00:51:12.029779 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:12.029751 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:12.267970 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:12.267874 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:12.894576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:12.894544 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 00:51:12.894972 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:12.894595 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 00:51:12.935974 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:12.935939 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:13.142697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:13.142668 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:13.894425 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:13.894395 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 00:51:13.894610 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:13.894445 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 00:51:14.731383 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:14.731338 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:14.893762 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:14.893720 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 00:51:14.893933 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:14.893805 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 00:51:14.957036 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:14.957003 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:15.285950 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.285911 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:15.511204 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.511090 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:15.692054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.691954 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89"] Nov 01 00:51:15.696756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.696733 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:15.740885 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.740850 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:15.755966 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.755936 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89"] Nov 01 00:51:15.803266 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.803235 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-64566096-dockercfg-6dwjf\"" Nov 01 00:51:15.804224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.804181 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-64566096\"" Nov 01 00:51:15.805926 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.805885 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89\" (UID: \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:15.807565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.807536 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9npg\" (UniqueName: \"kubernetes.io/projected/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-kube-api-access-v9npg\") pod \"payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89\" (UID: \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:15.851757 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.851695 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md"] Nov 01 00:51:15.856559 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.856532 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:15.909579 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.909532 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9npg\" (UniqueName: \"kubernetes.io/projected/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-kube-api-access-v9npg\") pod \"payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89\" (UID: \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:15.909792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.909619 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e804a6c-f36e-46ea-8a04-109911b25c56-plugins-config-volume\") pod \"payload-processing-e2e-ait-64566096-57b9674bb6-bc4md\" (UID: \"7e804a6c-f36e-46ea-8a04-109911b25c56\") " pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:15.909792 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.909736 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89\" (UID: \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:15.910060 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.910040 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpww4\" (UniqueName: \"kubernetes.io/projected/7e804a6c-f36e-46ea-8a04-109911b25c56-kube-api-access-lpww4\") pod \"payload-processing-e2e-ait-64566096-57b9674bb6-bc4md\" (UID: \"7e804a6c-f36e-46ea-8a04-109911b25c56\") " pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:15.911377 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.911341 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89\" (UID: \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:15.919458 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.919428 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md"] Nov 01 00:51:15.998609 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:15.996921 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9npg\" (UniqueName: \"kubernetes.io/projected/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-kube-api-access-v9npg\") pod \"payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89\" (UID: \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:16.018913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.016372 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e804a6c-f36e-46ea-8a04-109911b25c56-plugins-config-volume\") pod \"payload-processing-e2e-ait-64566096-57b9674bb6-bc4md\" (UID: \"7e804a6c-f36e-46ea-8a04-109911b25c56\") " pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:16.018913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.016567 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpww4\" (UniqueName: \"kubernetes.io/projected/7e804a6c-f36e-46ea-8a04-109911b25c56-kube-api-access-lpww4\") pod \"payload-processing-e2e-ait-64566096-57b9674bb6-bc4md\" (UID: \"7e804a6c-f36e-46ea-8a04-109911b25c56\") " pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:16.018913 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.018110 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e804a6c-f36e-46ea-8a04-109911b25c56-plugins-config-volume\") pod \"payload-processing-e2e-ait-64566096-57b9674bb6-bc4md\" (UID: \"7e804a6c-f36e-46ea-8a04-109911b25c56\") " pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:16.069311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.049617 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:16.095785 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.095752 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpww4\" (UniqueName: \"kubernetes.io/projected/7e804a6c-f36e-46ea-8a04-109911b25c56-kube-api-access-lpww4\") pod \"payload-processing-e2e-ait-64566096-57b9674bb6-bc4md\" (UID: \"7e804a6c-f36e-46ea-8a04-109911b25c56\") " pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:16.121375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.121267 6036 scope.go:122] "RemoveContainer" containerID="a9206863426fbf9cb69315228fe1e1afaf5c44fddc69e2a5f44b7980998abe7a" Nov 01 00:51:16.176946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.176915 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:16.295898 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.294638 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:16.296268 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.296246 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:16.296376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.296345 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:51:16.542357 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:51:16.542327 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3b391a_7a9d_4bb6_ae51_1efa84db6451.slice/crio-7fe137a5af1430e880cdf73e8df36ef8d97d6845ec0e9887f19c50ce9d5d098c WatchSource:0}: Error finding container 7fe137a5af1430e880cdf73e8df36ef8d97d6845ec0e9887f19c50ce9d5d098c: Status 404 returned error can't find the container with id 7fe137a5af1430e880cdf73e8df36ef8d97d6845ec0e9887f19c50ce9d5d098c Nov 01 00:51:16.542947 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.542919 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89"] Nov 01 00:51:16.598975 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.598946 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md"] Nov 01 00:51:16.603545 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:51:16.603522 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e804a6c_f36e_46ea_8a04_109911b25c56.slice/crio-20440e83f793aea11470faf02b2a85402d171489cb82e00d2531d932ffd08392 WatchSource:0}: Error finding container 20440e83f793aea11470faf02b2a85402d171489cb82e00d2531d932ffd08392: Status 404 returned error can't find the container with id 20440e83f793aea11470faf02b2a85402d171489cb82e00d2531d932ffd08392 Nov 01 00:51:16.910722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.910626 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" event={"ID":"7e804a6c-f36e-46ea-8a04-109911b25c56","Type":"ContainerStarted","Data":"54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b"} Nov 01 00:51:16.910722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.910685 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:16.910722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.910711 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" event={"ID":"7e804a6c-f36e-46ea-8a04-109911b25c56","Type":"ContainerStarted","Data":"20440e83f793aea11470faf02b2a85402d171489cb82e00d2531d932ffd08392"} Nov 01 00:51:16.912207 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.912180 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" event={"ID":"5d3b391a-7a9d-4bb6-ae51-1efa84db6451","Type":"ContainerStarted","Data":"04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0"} Nov 01 00:51:16.912361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.912219 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" event={"ID":"5d3b391a-7a9d-4bb6-ae51-1efa84db6451","Type":"ContainerStarted","Data":"7fe137a5af1430e880cdf73e8df36ef8d97d6845ec0e9887f19c50ce9d5d098c"} Nov 01 00:51:16.912361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:16.912239 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:17.029754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:17.029704 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" podStartSLOduration=2.029690053 podStartE2EDuration="2.029690053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:17.029368388 +0000 UTC m=+1621.904589402" watchObservedRunningTime="2026-11-01 00:51:17.029690053 +0000 UTC m=+1621.904911065" Nov 01 00:51:17.155983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:17.155919 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" podStartSLOduration=2.155901139 podStartE2EDuration="2.155901139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:17.152254455 +0000 UTC m=+1622.027475472" watchObservedRunningTime="2026-11-01 00:51:17.155901139 +0000 UTC m=+1622.031122156" Nov 01 00:51:18.257396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.257360 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:18.296890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.296860 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7"] Nov 01 00:51:18.301619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.301595 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390047 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390010 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390249 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390075 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390249 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390210 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390395 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390277 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390388 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390525 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390506 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e00ff82-0096-4553-9f58-43b412f61f30-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390668 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390652 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e00ff82-0096-4553-9f58-43b412f61f30-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390713 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390696 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.390808 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.390791 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4m7\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-kube-api-access-hq4m7\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.465150 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.465119 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:18.492095 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492057 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492125 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4m7\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-kube-api-access-hq4m7\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492202 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492258 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492335 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492385 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492402 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492487 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e00ff82-0096-4553-9f58-43b412f61f30-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492712 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492503 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492712 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492585 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492712 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492669 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e00ff82-0096-4553-9f58-43b412f61f30-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492732 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.492884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.492852 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.493457 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.493431 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e00ff82-0096-4553-9f58-43b412f61f30-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.494984 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.494954 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.495327 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.495269 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e00ff82-0096-4553-9f58-43b412f61f30-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.507563 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.507500 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7"] Nov 01 00:51:18.555512 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.555478 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.556767 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.556742 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4m7\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-kube-api-access-hq4m7\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:18.617637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:18.617603 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:19.051812 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:19.051781 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7"] Nov 01 00:51:19.052511 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:51:19.052486 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e00ff82_0096_4553_9f58_43b412f61f30.slice/crio-dd42e8157dfaec6f16f89e5bc90206bc962316701ff53e2e47256475e5e5eb01 WatchSource:0}: Error finding container dd42e8157dfaec6f16f89e5bc90206bc962316701ff53e2e47256475e5e5eb01: Status 404 returned error can't find the container with id dd42e8157dfaec6f16f89e5bc90206bc962316701ff53e2e47256475e5e5eb01 Nov 01 00:51:19.055321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:19.055271 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:19.055389 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:19.055354 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:19.055389 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:19.055384 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:19.511061 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:51:19.510538 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:51:19.935567 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:19.935530 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" event={"ID":"2e00ff82-0096-4553-9f58-43b412f61f30","Type":"ContainerStarted","Data":"85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3"} Nov 01 00:51:19.935757 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:19.935586 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" event={"ID":"2e00ff82-0096-4553-9f58-43b412f61f30","Type":"ContainerStarted","Data":"dd42e8157dfaec6f16f89e5bc90206bc962316701ff53e2e47256475e5e5eb01"} Nov 01 00:51:20.112093 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:20.112029 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podStartSLOduration=2.112006843 podStartE2EDuration="2.112006843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:20.109830076 +0000 UTC m=+1624.985051092" watchObservedRunningTime="2026-11-01 00:51:20.112006843 +0000 UTC m=+1624.987227858" Nov 01 00:51:20.144809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:20.144778 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:20.618680 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:20.618650 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:20.620043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:20.620019 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 00:51:20.620108 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:20.620074 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 00:51:21.618094 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:21.618056 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 00:51:21.618331 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:21.618122 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 00:51:22.384737 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.384700 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9"] Nov 01 00:51:22.389575 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.389554 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.482348 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.481181 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9"] Nov 01 00:51:22.486590 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486562 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtz8\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-kube-api-access-bmtz8\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.486770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486627 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.486770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486767 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.486907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486805 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094e131e-acdd-4ef3-ae8f-a499884df745-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.486907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486840 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.486907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486860 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.486907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486888 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.486907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486909 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.487085 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.486925 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094e131e-acdd-4ef3-ae8f-a499884df745-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.494647 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.494613 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:22.588096 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588063 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtz8\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-kube-api-access-bmtz8\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588124 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588216 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588247 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094e131e-acdd-4ef3-ae8f-a499884df745-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588278 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588423 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588468 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588497 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588536 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588517 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094e131e-acdd-4ef3-ae8f-a499884df745-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588796 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588756 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588879 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588818 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588879 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588858 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.588999 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.588976 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.589313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.589264 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094e131e-acdd-4ef3-ae8f-a499884df745-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.590952 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.590927 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.591059 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.591000 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094e131e-acdd-4ef3-ae8f-a499884df745-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.618602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.618574 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 00:51:22.618750 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.618632 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 00:51:22.671148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.671061 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.674399 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.674332 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtz8\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-kube-api-access-bmtz8\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.781328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.739723 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:22.781328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.741891 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:22.989131 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:22.989089 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:23.136423 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.136383 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl"] Nov 01 00:51:23.219219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.211768 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:23.219219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.218903 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl"] Nov 01 00:51:23.276315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.272470 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-plugins-config-volume\") pod \"payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl\" (UID: \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\") " pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:23.276315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.272585 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr67\" (UniqueName: \"kubernetes.io/projected/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-kube-api-access-lnr67\") pod \"payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl\" (UID: \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\") " pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:23.284945 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.284917 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8a74f100\"" Nov 01 00:51:23.285901 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.285609 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8a74f100-dockercfg-sh79k\"" Nov 01 00:51:23.390842 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.388253 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-plugins-config-volume\") pod \"payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl\" (UID: \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\") " pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:23.390842 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.388396 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr67\" (UniqueName: \"kubernetes.io/projected/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-kube-api-access-lnr67\") pod \"payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl\" (UID: \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\") " pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:23.405700 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.398641 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-plugins-config-volume\") pod \"payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl\" (UID: \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\") " pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:23.415957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.415904 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:23.416128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.416004 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:23.416128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.416051 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:23.418504 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.418479 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9"] Nov 01 00:51:23.475548 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.475513 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr67\" (UniqueName: \"kubernetes.io/projected/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-kube-api-access-lnr67\") pod \"payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl\" (UID: \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\") " pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:23.544184 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.544156 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:23.627370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.624715 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 00:51:23.627370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:23.624767 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 00:51:24.013943 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.013906 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" event={"ID":"094e131e-acdd-4ef3-ae8f-a499884df745","Type":"ContainerStarted","Data":"b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15"} Nov 01 00:51:24.015537 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.015461 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" event={"ID":"094e131e-acdd-4ef3-ae8f-a499884df745","Type":"ContainerStarted","Data":"f36306f0ee2e7937b5241413d29afb86c45ea5e234e10fd1df35a8bb72e0419b"} Nov 01 00:51:24.145156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.145110 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podStartSLOduration=2.1450969779999998 podStartE2EDuration="2.145096978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:24.1440462 +0000 UTC m=+1629.019267216" watchObservedRunningTime="2026-11-01 00:51:24.145096978 +0000 UTC m=+1629.020317992" Nov 01 00:51:24.275858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.275801 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl"] Nov 01 00:51:24.282964 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:51:24.282934 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b44ac67_5d41_4d1d_b378_211c5d1b08e1.slice/crio-f36ac01e77ab749334dcf49264a7c0b2fb736c771df70160ccc77cb0e9029f84 WatchSource:0}: Error finding container f36ac01e77ab749334dcf49264a7c0b2fb736c771df70160ccc77cb0e9029f84: Status 404 returned error can't find the container with id f36ac01e77ab749334dcf49264a7c0b2fb736c771df70160ccc77cb0e9029f84 Nov 01 00:51:24.618071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.618045 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 00:51:24.618527 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.618093 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 00:51:24.740600 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.740567 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:24.741925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.741893 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 00:51:24.742055 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:24.741949 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 00:51:25.022078 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.021984 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" event={"ID":"0b44ac67-5d41-4d1d-b378-211c5d1b08e1","Type":"ContainerStarted","Data":"1fa05f29727f48a3ba19203269c6455cdb37ed6b3b6c637c7e0e1cb076af9718"} Nov 01 00:51:25.022336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.022314 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" event={"ID":"0b44ac67-5d41-4d1d-b378-211c5d1b08e1","Type":"ContainerStarted","Data":"f36ac01e77ab749334dcf49264a7c0b2fb736c771df70160ccc77cb0e9029f84"} Nov 01 00:51:25.022479 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.022466 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:25.178416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.178334 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" podStartSLOduration=2.178269076 podStartE2EDuration="2.178269076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:25.16547236 +0000 UTC m=+1630.040693403" watchObservedRunningTime="2026-11-01 00:51:25.178269076 +0000 UTC m=+1630.053490090" Nov 01 00:51:25.286812 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.286778 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:25.618874 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.618777 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 00:51:25.618874 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.618847 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 00:51:25.741270 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.741237 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 00:51:25.741494 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:25.741312 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 00:51:26.740979 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:26.740947 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 00:51:26.741584 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:26.741001 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 00:51:26.827905 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:26.827707 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:26.829102 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:26.828767 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:26.831786 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:26.831763 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 00:51:27.741437 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:27.741402 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 00:51:27.741943 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:27.741479 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 00:51:27.922941 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:27.922912 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:51:27.923112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:27.922981 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:51:28.740775 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:28.740737 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 00:51:28.740948 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:28.740810 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 00:51:29.000114 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:29.000019 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:29.211300 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:29.211261 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:29.583424 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:29.583393 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:29.740675 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:29.740640 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 00:51:29.740852 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:29.740709 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 00:51:30.397941 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:30.397903 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:30.740488 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:30.740394 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 00:51:30.740488 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:30.740471 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 00:51:31.745714 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:31.745683 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:31.746174 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:31.746009 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:31.747447 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:31.747421 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:51:34.764561 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:34.764533 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:34.974271 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:34.974241 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:35.188641 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:35.188603 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:35.430195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:35.430162 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:35.834324 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:35.831740 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:36.039097 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:36.039019 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:51:36.068043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:36.068012 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:38.044187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.043419 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:38.142550 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.141412 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.169442 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.169388 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.203727 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.202813 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.260311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.259958 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.333629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.333405 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:51:38.355504 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.355437 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.396508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.396443 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 00:51:38.414589 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.414533 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.502325 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.496399 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.566950 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.566910 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.776052 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.775958 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:51:38.819361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.819318 6036 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:38.819644 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.819614 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:38.871759 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.871717 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 00:51:38.967440 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:38.967389 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.007314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.003856 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:51:39.044433 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.044386 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.045475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.045453 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:51:39.143441 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.143391 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.145219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.145190 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:51:39.208470 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.208417 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.209162 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.209139 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:51:39.451608 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.451508 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:51:39.452137 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.452104 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.457919 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.457891 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:51:39.458152 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.458134 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:51:39.581715 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.581662 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.585570 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.582958 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:51:39.607387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.582990 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:51:39.607387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.595446 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.988047 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.987998 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:51:39.991451 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:39.991407 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 00:51:40.012323 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.004717 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:51:40.037408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.037262 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 01 00:51:40.137209 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.110731 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:40.285376 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.260508 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:40.556515 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.343871 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:40.580315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.430273 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:51:40.580315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.567218 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:51:40.580315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.567320 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:51:40.580315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.567474 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:51:40.580315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.567540 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:51:40.580315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.567599 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:51:40.580315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.567746 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:51:40.610357 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.495083 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Nov 01 00:51:40.621398 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.495221 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:51:40.627748 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.627685 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 00:51:40.656709 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.495354 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 00:51:40.708350 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.656953 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:51:40.745484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.659557 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:40.791514 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.791453 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:40.799369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:40.795442 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.058923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.058867 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.061051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.061017 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:51:41.061469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.061446 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 01 00:51:41.084315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.081002 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:51:41.132309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.113733 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.132813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.115969 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 00:51:41.327507 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.324918 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:41.470852 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.470812 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.523339 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.523280 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:41.523676 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.523627 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.577629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.576643 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:51:41.663865 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.663801 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.680427 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.680275 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.725316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.714436 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 00:51:41.841374 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.821312 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:51:42.451122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.843334 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Nov 01 00:51:42.451840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:41.909569 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.460349 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.262354 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 01 00:51:42.460758 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.324949 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:51:42.485071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.324984 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:42.485606 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.325005 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.485980 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.367852 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 00:51:42.486442 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.367893 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:51:42.486797 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.387417 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.487165 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.462004 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:42.658942 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.657572 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:51:42.687899 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.685058 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:51:42.687899 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.687109 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.715454 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.712055 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:42.739340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.719434 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.827398 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.826669 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7f5bn" podUID="e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 01 00:51:42.930329 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:51:42.923042 6036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.659s" Nov 01 00:51:42.958213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.955441 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:42.958213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:42.955511 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.473309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.463185 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:51:43.473309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.463592 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:43.485389 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.485331 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 01 00:51:43.486309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.486202 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.487853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.487161 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:43.487853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.487354 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.488045 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.487992 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.488871 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.488152 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 01 00:51:43.518340 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.511944 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:43.674812 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.674755 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 00:51:43.862708 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.687932 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.891784 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.710754 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.892741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.710805 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:43.892741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.892391 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.892741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:43.834836 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 00:51:44.055312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.051314 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:44.146489 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.146396 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:44.146756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.146723 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.302256 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.302167 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:44.362516 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.362486 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 00:51:44.383370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.382322 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 00:51:44.465820 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.465503 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.468479 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.466242 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 00:51:44.491488 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.491433 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.493782 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.493499 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.494390 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.494239 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.495460 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.495425 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.507821 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.506687 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 00:51:44.512840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.512807 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.531220 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.528413 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 00:51:44.533244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.533223 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 00:51:44.534718 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.534699 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 00:51:44.558550 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.558247 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 00:51:44.567184 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.567154 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" Nov 01 00:51:44.593547 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.593522 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 00:51:44.602973 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.602947 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 00:51:44.630346 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.622923 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:51:44.657435 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.657212 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:44.659725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:44.659696 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 00:51:45.096819 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:45.096787 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:45.330974 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:45.330937 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:47.299795 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:47.299760 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:49.325147 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:49.325112 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:49.506544 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:51:49.506506 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:51:50.156146 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:50.156114 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:55.048120 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:55.048071 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:51:57.028902 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.028862 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm"] Nov 01 00:51:57.034108 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.034088 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.079800 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.079765 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn"] Nov 01 00:51:57.084251 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.084232 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.091587 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.091564 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srhv5\"" Nov 01 00:51:57.103821 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.103787 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.103959 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.103934 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.104019 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.103977 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.104019 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.104003 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.104101 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.104089 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.104191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.104177 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.104250 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.104235 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbms\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-kube-api-access-gvbms\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.104329 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.104311 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.104381 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.104367 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.140416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.140373 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm"] Nov 01 00:51:57.188123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.188090 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn"] Nov 01 00:51:57.205869 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.205832 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.205882 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.205933 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.206040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.205981 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206228 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206200 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.206342 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206275 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206425 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206352 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtg2\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-kube-api-access-mhtg2\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.206487 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206425 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbms\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-kube-api-access-gvbms\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206540 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206484 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206540 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206488 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206668 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206648 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da902477-5e58-4752-b788-4488ce5821c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.206729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206695 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.206785 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206732 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206785 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206752 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206785 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206732 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.206938 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206862 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.206938 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.206909 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.207065 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.207047 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.207117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.207072 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.207117 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.207092 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.207215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.207140 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.207215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.207192 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da902477-5e58-4752-b788-4488ce5821c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.207357 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.207337 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.209719 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.209687 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.209935 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.209914 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.271465 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.271432 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbms\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-kube-api-access-gvbms\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.271877 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.271853 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-94qbm\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.308882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.308826 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.309085 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.308913 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtg2\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-kube-api-access-mhtg2\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.309085 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.309073 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da902477-5e58-4752-b788-4488ce5821c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.309224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.309111 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.309224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.309149 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.309224 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.309204 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.309407 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.309386 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.309469 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.309442 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da902477-5e58-4752-b788-4488ce5821c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.310756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.309589 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.310756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.310249 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.310756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.310360 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da902477-5e58-4752-b788-4488ce5821c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.310756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.310519 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.311067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.310903 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.311067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.310973 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.312706 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.312681 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.313456 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.313430 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da902477-5e58-4752-b788-4488ce5821c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.349593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.349554 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:57.373354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.373264 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.373471 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.373412 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtg2\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-kube-api-access-mhtg2\") pod \"maas-default-gateway-openshift-default-67bf986f99-dd6kn\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.398680 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.398646 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:57.790218 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:51:57.790185 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaec49be_2da6_47ae_aad3_9ec0a797c6f8.slice/crio-9155cbb046420f6990462dd914bbd4c5fbd2442a81859dc5dcb79cab0523281d WatchSource:0}: Error finding container 9155cbb046420f6990462dd914bbd4c5fbd2442a81859dc5dcb79cab0523281d: Status 404 returned error can't find the container with id 9155cbb046420f6990462dd914bbd4c5fbd2442a81859dc5dcb79cab0523281d Nov 01 00:51:57.792127 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.792098 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm"] Nov 01 00:51:57.792806 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.792773 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:57.792859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.792841 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:57.792895 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.792870 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:57.842967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.842936 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn"] Nov 01 00:51:57.843117 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:51:57.842974 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda902477_5e58_4752_b788_4488ce5821c5.slice/crio-6bddd55ab7c7b45ee2b3da32feff97b778814e2d02709e6d4e5832cb7dd53258 WatchSource:0}: Error finding container 6bddd55ab7c7b45ee2b3da32feff97b778814e2d02709e6d4e5832cb7dd53258: Status 404 returned error can't find the container with id 6bddd55ab7c7b45ee2b3da32feff97b778814e2d02709e6d4e5832cb7dd53258 Nov 01 00:51:57.846153 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.846120 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:57.846229 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.846206 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:57.846262 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:57.846236 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:58.387466 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.387434 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" event={"ID":"da902477-5e58-4752-b788-4488ce5821c5","Type":"ContainerStarted","Data":"f3832572f819b903cde7180fd971b74fc592e028caeac4984a0de2f029efa98b"} Nov 01 00:51:58.387900 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.387500 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" event={"ID":"da902477-5e58-4752-b788-4488ce5821c5","Type":"ContainerStarted","Data":"6bddd55ab7c7b45ee2b3da32feff97b778814e2d02709e6d4e5832cb7dd53258"} Nov 01 00:51:58.389332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.389267 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" event={"ID":"caec49be-2da6-47ae-aad3-9ec0a797c6f8","Type":"ContainerStarted","Data":"a86da17d206b1742d78e7bf96d5bc26dda443b7011cb80f69bd854e4313c9fe0"} Nov 01 00:51:58.389332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.389332 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" event={"ID":"caec49be-2da6-47ae-aad3-9ec0a797c6f8","Type":"ContainerStarted","Data":"9155cbb046420f6990462dd914bbd4c5fbd2442a81859dc5dcb79cab0523281d"} Nov 01 00:51:58.399742 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.399715 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:51:58.401254 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.401227 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 00:51:58.401449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.401274 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 00:51:58.514854 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.514801 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podStartSLOduration=2.514785932 podStartE2EDuration="2.514785932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:58.511226689 +0000 UTC m=+1663.386447703" watchObservedRunningTime="2026-11-01 00:51:58.514785932 +0000 UTC m=+1663.390006947" Nov 01 00:51:58.694369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:58.679275 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podStartSLOduration=2.679254648 podStartE2EDuration="2.679254648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:58.645218244 +0000 UTC m=+1663.520439259" watchObservedRunningTime="2026-11-01 00:51:58.679254648 +0000 UTC m=+1663.554475663" Nov 01 00:51:59.359310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:59.357678 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:51:59.383310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:59.362851 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-94qbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 00:51:59.383310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:59.362908 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 00:51:59.400031 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:59.399901 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 00:51:59.400031 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:59.399978 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 00:51:59.808221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:51:59.808163 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:00.052315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:00.051562 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:00.294719 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:00.291797 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:00.365317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:00.358439 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-94qbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 00:52:00.365317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:00.358516 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 00:52:00.399990 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:00.399951 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 00:52:00.400346 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:00.400314 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 00:52:01.368825 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:01.368790 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-94qbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 00:52:01.369150 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:01.369120 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 00:52:01.399658 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:01.399621 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 00:52:01.399858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:01.399682 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 00:52:02.353763 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:02.353723 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-94qbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 00:52:02.354407 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:02.354371 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 00:52:02.399483 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:02.399446 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 00:52:02.399666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:02.399501 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 00:52:03.390434 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:03.390384 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-94qbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 00:52:03.391307 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:03.391011 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 00:52:03.400311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:03.400045 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 00:52:03.400711 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:03.400676 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 00:52:04.349902 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:04.349871 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-94qbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 00:52:04.350112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:04.349939 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 00:52:04.400168 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:04.400128 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 00:52:04.401403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:04.400194 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 00:52:04.892930 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:04.892893 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:05.104123 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:05.104087 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:05.350906 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:05.350870 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-94qbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 00:52:05.351099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:05.350934 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 00:52:05.399502 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:05.399460 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 00:52:05.399672 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:05.399546 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 00:52:06.776765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:06.776726 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:52:06.777897 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:06.777869 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:52:06.778387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:06.778346 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 00:52:06.793703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:06.793677 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:52:06.794458 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:06.794435 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:52:06.797107 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:06.797082 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 00:52:09.878072 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:09.878040 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:12.150492 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.150459 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr"] Nov 01 00:52:12.155345 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.155325 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.258982 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.258946 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr"] Nov 01 00:52:12.264700 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.264663 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l52s\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-kube-api-access-4l52s\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.264871 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.264842 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.264933 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.264888 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.264994 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.264956 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.264994 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.264987 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.265128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.265109 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.265667 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.265168 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.265667 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.265212 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.265667 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.265246 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.366725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.366687 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l52s\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-kube-api-access-4l52s\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.366903 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.366795 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.366903 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.366817 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.366903 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.366870 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367032 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367000 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367197 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367327 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367266 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367485 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367466 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367271 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367310 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367678 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367559 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367799 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367771 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367799 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367788 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.367948 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.367926 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.369566 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.369545 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.369770 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.369750 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.430214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.430117 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.430403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.430226 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l52s\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-kube-api-access-4l52s\") pod \"maas-default-gateway-openshift-default-67bf986f99-h6nmr\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.471272 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.471234 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:12.911431 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.911401 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr"] Nov 01 00:52:12.912005 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:52:12.911977 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdd8a31_13ec_4bdf_aa71_b6d815af2dd2.slice/crio-acd84377844dfd3cb6e97352a9e3d27b01392db4a156c7d7aa59704660f0d328 WatchSource:0}: Error finding container acd84377844dfd3cb6e97352a9e3d27b01392db4a156c7d7aa59704660f0d328: Status 404 returned error can't find the container with id acd84377844dfd3cb6e97352a9e3d27b01392db4a156c7d7aa59704660f0d328 Nov 01 00:52:12.914842 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.914806 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:12.914932 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.914874 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:12.914932 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:12.914904 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:13.516203 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:13.516169 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" event={"ID":"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2","Type":"ContainerStarted","Data":"504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e"} Nov 01 00:52:13.516666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:13.516217 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" event={"ID":"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2","Type":"ContainerStarted","Data":"acd84377844dfd3cb6e97352a9e3d27b01392db4a156c7d7aa59704660f0d328"} Nov 01 00:52:13.639910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:13.639844 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podStartSLOduration=1.63982706 podStartE2EDuration="1.63982706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:13.638237234 +0000 UTC m=+1678.513458248" watchObservedRunningTime="2026-11-01 00:52:13.63982706 +0000 UTC m=+1678.515048075" Nov 01 00:52:14.471791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:14.471742 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:14.473257 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:14.473232 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 00:52:14.473415 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:14.473297 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 00:52:14.762369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:14.760061 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:15.482313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:15.476158 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 00:52:15.482313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:15.476218 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 00:52:16.472048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:16.472002 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 00:52:16.472603 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:16.472054 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 00:52:17.472507 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:17.472458 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 00:52:17.473071 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:17.472527 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 00:52:18.471873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:18.471837 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 00:52:18.472061 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:18.471894 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 00:52:19.471779 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:19.471737 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 00:52:19.472199 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:19.471812 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 00:52:19.512859 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:52:19.512826 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:52:19.835716 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:19.835649 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:20.472619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:20.472590 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 00:52:20.473194 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:20.473154 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 00:52:21.475310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:21.474760 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 00:52:21.475310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:21.474836 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 00:52:23.497079 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:23.497038 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:23.527856 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:23.527673 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:52:23.527856 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:23.527738 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:23.718382 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:23.718346 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:23.719933 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:23.719895 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 01 00:52:23.744731 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:23.742667 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:23.761188 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:23.760035 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 00:52:24.727473 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:24.727426 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:24.728629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:24.728604 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:52:24.936396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:24.935996 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 00:52:25.112309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:25.112232 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:25.318447 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:25.318412 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:25.542280 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:25.542182 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:25.766314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:25.759937 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:27.879466 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:27.879424 6036 patch_prober.go:29] interesting pod/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:52:27.879923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:27.879515 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:29.945234 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:29.945195 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:30.648478 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:30.648441 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:30.754031 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:30.753999 6036 generic.go:356] "Generic (PLEG): container finished" podID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerID="e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676" exitCode=0 Nov 01 00:52:30.754205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:30.754075 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" event={"ID":"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d","Type":"ContainerDied","Data":"e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676"} Nov 01 00:52:30.754597 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:30.754583 6036 scope.go:122] "RemoveContainer" containerID="e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676" Nov 01 00:52:30.756457 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:30.756428 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:30.756549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:30.756498 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:30.756549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:30.756533 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:31.670309 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:31.667085 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:31.781019 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:31.780982 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" event={"ID":"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d","Type":"ContainerStarted","Data":"98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa"} Nov 01 00:52:31.832118 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:31.832088 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:52:31.832279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:31.832260 6036 patch_prober.go:29] interesting pod/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 00:52:31.832384 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:31.832331 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 00:52:31.910753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:31.908812 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:32.832639 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:32.832604 6036 patch_prober.go:29] interesting pod/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 00:52:32.833034 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:32.832661 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 00:52:33.831963 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:33.831929 6036 patch_prober.go:29] interesting pod/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 00:52:33.832174 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:33.831991 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 00:52:34.832415 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:34.832380 6036 patch_prober.go:29] interesting pod/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 00:52:34.832798 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:34.832431 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 00:52:34.975890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:34.975853 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6"] Nov 01 00:52:34.980333 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:34.980275 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.028528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.028500 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6"] Nov 01 00:52:35.033990 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.033960 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-14957a74-serving-cert\"" Nov 01 00:52:35.100396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.098577 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.100396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.098666 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffs4\" (UniqueName: \"kubernetes.io/projected/0ca7d99e-855e-4290-8e64-5efe096b5c0d-kube-api-access-2ffs4\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.104474 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.104100 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca7d99e-855e-4290-8e64-5efe096b5c0d-openshift-service-ca\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.104474 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.104204 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-metrics-certs\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.206603 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.206568 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca7d99e-855e-4290-8e64-5efe096b5c0d-openshift-service-ca\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.206783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.206632 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-metrics-certs\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.206783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.206763 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.206878 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.206827 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffs4\" (UniqueName: \"kubernetes.io/projected/0ca7d99e-855e-4290-8e64-5efe096b5c0d-kube-api-access-2ffs4\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.206997 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:52:35.206968 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-serving-cert: secret "maas-api-e2e-migrate-14957a74-serving-cert" not found Nov 01 00:52:35.208039 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:52:35.207053 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls podName:0ca7d99e-855e-4290-8e64-5efe096b5c0d nodeName:}" failed. No retries permitted until 2026-11-01 00:52:35.707031792 +0000 UTC m=+1700.582252805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls") pod "maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" (UID: "0ca7d99e-855e-4290-8e64-5efe096b5c0d") : secret "maas-api-e2e-migrate-14957a74-serving-cert" not found Nov 01 00:52:35.208766 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.208743 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca7d99e-855e-4290-8e64-5efe096b5c0d-openshift-service-ca\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.212109 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.212081 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-metrics-certs\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.269314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.268170 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:35.291916 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.291892 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffs4\" (UniqueName: \"kubernetes.io/projected/0ca7d99e-855e-4290-8e64-5efe096b5c0d-kube-api-access-2ffs4\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.490563 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.490461 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:35.703247 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.703210 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:35.718137 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.718108 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.720803 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.720783 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls\") pod \"maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:35.831824 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.831789 6036 patch_prober.go:29] interesting pod/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 00:52:35.832007 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.831855 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 00:52:35.899932 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:35.899897 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:36.315031 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:36.314992 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6"] Nov 01 00:52:36.315745 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:52:36.315720 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca7d99e_855e_4290_8e64_5efe096b5c0d.slice/crio-288630052aae97fdce8a7ea755964fcadcc2c0f8b45485dac7c3e5c3239e1093 WatchSource:0}: Error finding container 288630052aae97fdce8a7ea755964fcadcc2c0f8b45485dac7c3e5c3239e1093: Status 404 returned error can't find the container with id 288630052aae97fdce8a7ea755964fcadcc2c0f8b45485dac7c3e5c3239e1093 Nov 01 00:52:36.317219 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:36.317201 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:52:36.821451 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:36.821415 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" event={"ID":"0ca7d99e-855e-4290-8e64-5efe096b5c0d","Type":"ContainerStarted","Data":"288630052aae97fdce8a7ea755964fcadcc2c0f8b45485dac7c3e5c3239e1093"} Nov 01 00:52:36.831896 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:36.831868 6036 patch_prober.go:29] interesting pod/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 00:52:36.832085 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:36.831924 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 00:52:37.830119 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:37.830088 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" event={"ID":"0ca7d99e-855e-4290-8e64-5efe096b5c0d","Type":"ContainerStarted","Data":"878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3"} Nov 01 00:52:37.830549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:37.830152 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:37.836783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:37.836756 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:52:37.837075 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:37.837056 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:52:37.838013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:37.837994 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:52:37.951208 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:37.951152 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" podStartSLOduration=3.163145607 podStartE2EDuration="3.951138233s" totalImagesPullingTime="787.992626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:37.950569815 +0000 UTC m=+1702.825790830" watchObservedRunningTime="2026-11-01 00:52:37.951138233 +0000 UTC m=+1702.826359247" Nov 01 00:52:40.032664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:40.030387 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:43.169856 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:43.169818 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:43.844017 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:43.843986 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:52:45.038870 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:45.038829 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:49.449648 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:49.449609 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:49.509306 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:52:49.509255 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:52:49.878023 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:49.877988 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:54.750029 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:54.749938 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:52:59.491441 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:59.491387 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx" podUID="01758a46-b41c-4ac1-b4b7-6baf086384e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 00:52:59.493230 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:59.493194 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:59.641255 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:59.641199 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:59.713312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:59.711407 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:59.735445 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:59.727434 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:59.774408 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:52:59.773920 6036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:00.161056 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:00.161017 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:05.056793 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:05.056758 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:05.369431 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:05.369343 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:06.361103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:06.361062 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:12.407184 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:12.407143 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:15.982778 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:15.982739 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:17.905489 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:17.905442 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:18.349637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:18.349602 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md"] Nov 01 00:53:18.437597 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:18.437552 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89"] Nov 01 00:53:18.826064 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:18.826022 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:19.095028 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:19.094927 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:19.508127 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:53:19.508035 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:53:23.161864 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.161813 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:23.350915 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.350870 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" podUID="7e804a6c-f36e-46ea-8a04-109911b25c56" containerName="payload-processing" containerID="cri-o://54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b" gracePeriod=25 Nov 01 00:53:23.437929 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.437822 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" podUID="5d3b391a-7a9d-4bb6-ae51-1efa84db6451" containerName="payload-pre-processing" containerID="cri-o://04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0" gracePeriod=25 Nov 01 00:53:23.733894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.733863 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:53:23.795079 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.795046 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:53:23.807005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.806977 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e804a6c-f36e-46ea-8a04-109911b25c56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e804a6c-f36e-46ea-8a04-109911b25c56-plugins-config-volume\") pod \"7e804a6c-f36e-46ea-8a04-109911b25c56\" (UID: \"7e804a6c-f36e-46ea-8a04-109911b25c56\") " Nov 01 00:53:23.807148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.807048 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e804a6c-f36e-46ea-8a04-109911b25c56-kube-api-access-lpww4\" (UniqueName: \"kubernetes.io/projected/7e804a6c-f36e-46ea-8a04-109911b25c56-kube-api-access-lpww4\") pod \"7e804a6c-f36e-46ea-8a04-109911b25c56\" (UID: \"7e804a6c-f36e-46ea-8a04-109911b25c56\") " Nov 01 00:53:23.807366 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.807345 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e804a6c-f36e-46ea-8a04-109911b25c56-plugins-config-volume" pod "7e804a6c-f36e-46ea-8a04-109911b25c56" (UID: "7e804a6c-f36e-46ea-8a04-109911b25c56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:53:23.807746 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.807730 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e804a6c-f36e-46ea-8a04-109911b25c56-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:23.809344 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.809325 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e804a6c-f36e-46ea-8a04-109911b25c56-kube-api-access-lpww4" pod "7e804a6c-f36e-46ea-8a04-109911b25c56" (UID: "7e804a6c-f36e-46ea-8a04-109911b25c56"). InnerVolumeSpecName "kube-api-access-lpww4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:23.908890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.908859 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-plugins-config-volume\") pod \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\" (UID: \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\") " Nov 01 00:53:23.909116 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.909037 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-kube-api-access-v9npg\" (UniqueName: \"kubernetes.io/projected/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-kube-api-access-v9npg\") pod \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\" (UID: \"5d3b391a-7a9d-4bb6-ae51-1efa84db6451\") " Nov 01 00:53:23.909369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.909341 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-plugins-config-volume" pod "5d3b391a-7a9d-4bb6-ae51-1efa84db6451" (UID: "5d3b391a-7a9d-4bb6-ae51-1efa84db6451"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:53:23.909843 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.909824 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpww4\" (UniqueName: \"kubernetes.io/projected/7e804a6c-f36e-46ea-8a04-109911b25c56-kube-api-access-lpww4\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:23.909843 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.909848 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:23.911484 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:23.911456 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-kube-api-access-v9npg" pod "5d3b391a-7a9d-4bb6-ae51-1efa84db6451" (UID: "5d3b391a-7a9d-4bb6-ae51-1efa84db6451"). InnerVolumeSpecName "kube-api-access-v9npg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:24.011424 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.011331 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9npg\" (UniqueName: \"kubernetes.io/projected/5d3b391a-7a9d-4bb6-ae51-1efa84db6451-kube-api-access-v9npg\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:24.155647 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.155599 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl"] Nov 01 00:53:24.345189 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.345156 6036 generic.go:356] "Generic (PLEG): container finished" podID="7e804a6c-f36e-46ea-8a04-109911b25c56" containerID="54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b" exitCode=0 Nov 01 00:53:24.345697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.345243 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" event={"ID":"7e804a6c-f36e-46ea-8a04-109911b25c56","Type":"ContainerDied","Data":"54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b"} Nov 01 00:53:24.345697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.345317 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" event={"ID":"7e804a6c-f36e-46ea-8a04-109911b25c56","Type":"ContainerDied","Data":"20440e83f793aea11470faf02b2a85402d171489cb82e00d2531d932ffd08392"} Nov 01 00:53:24.345697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.345326 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md" Nov 01 00:53:24.345697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.345344 6036 scope.go:122] "RemoveContainer" containerID="54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b" Nov 01 00:53:24.348883 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.348857 6036 generic.go:356] "Generic (PLEG): container finished" podID="5d3b391a-7a9d-4bb6-ae51-1efa84db6451" containerID="04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0" exitCode=0 Nov 01 00:53:24.349016 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.348900 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" event={"ID":"5d3b391a-7a9d-4bb6-ae51-1efa84db6451","Type":"ContainerDied","Data":"04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0"} Nov 01 00:53:24.349016 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.348922 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" Nov 01 00:53:24.349016 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.348935 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89" event={"ID":"5d3b391a-7a9d-4bb6-ae51-1efa84db6451","Type":"ContainerDied","Data":"7fe137a5af1430e880cdf73e8df36ef8d97d6845ec0e9887f19c50ce9d5d098c"} Nov 01 00:53:24.358417 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.358241 6036 scope.go:122] "RemoveContainer" containerID="54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b" Nov 01 00:53:24.358588 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:53:24.358569 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b\": container with ID starting with 54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b not found: ID does not exist" containerID="54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b" Nov 01 00:53:24.358638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.358597 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b"} err="failed to get container status \"54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b\": rpc error: code = NotFound desc = could not find container \"54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b\": container with ID starting with 54b9ba418fd88abf47f4acccff95e7c9152d310965accebe2c1eac71bfc8661b not found: ID does not exist" Nov 01 00:53:24.358638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.358616 6036 scope.go:122] "RemoveContainer" containerID="04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0" Nov 01 00:53:24.387785 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.387761 6036 scope.go:122] "RemoveContainer" containerID="04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0" Nov 01 00:53:24.388130 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:53:24.388109 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0\": container with ID starting with 04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0 not found: ID does not exist" containerID="04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0" Nov 01 00:53:24.388187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.388141 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0"} err="failed to get container status \"04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0\": rpc error: code = NotFound desc = could not find container \"04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0\": container with ID starting with 04396b2787f061f55badca2f4c1ae1fe2e461d32bd4eaf9250c01e16531587b0 not found: ID does not exist" Nov 01 00:53:24.517662 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.517629 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89"] Nov 01 00:53:24.521216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.521190 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64566096-75975f5cf5-csw89"] Nov 01 00:53:24.581054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.581023 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:24.636055 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.635965 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md"] Nov 01 00:53:24.640842 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.640815 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64566096-57b9674bb6-bc4md"] Nov 01 00:53:24.798561 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:24.798526 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:25.055745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:25.055710 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:25.253010 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:25.252973 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3b391a-7a9d-4bb6-ae51-1efa84db6451" path="/var/lib/kubelet/pods/5d3b391a-7a9d-4bb6-ae51-1efa84db6451/volumes" Nov 01 00:53:25.253514 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:25.253500 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e804a6c-f36e-46ea-8a04-109911b25c56" path="/var/lib/kubelet/pods/7e804a6c-f36e-46ea-8a04-109911b25c56/volumes" Nov 01 00:53:28.043133 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:28.043088 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:29.156993 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.156923 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" podUID="0b44ac67-5d41-4d1d-b378-211c5d1b08e1" containerName="payload-processing" containerID="cri-o://1fa05f29727f48a3ba19203269c6455cdb37ed6b3b6c637c7e0e1cb076af9718" gracePeriod=25 Nov 01 00:53:29.389363 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.389186 6036 generic.go:356] "Generic (PLEG): container finished" podID="0b44ac67-5d41-4d1d-b378-211c5d1b08e1" containerID="1fa05f29727f48a3ba19203269c6455cdb37ed6b3b6c637c7e0e1cb076af9718" exitCode=0 Nov 01 00:53:29.389363 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.389263 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" event={"ID":"0b44ac67-5d41-4d1d-b378-211c5d1b08e1","Type":"ContainerDied","Data":"1fa05f29727f48a3ba19203269c6455cdb37ed6b3b6c637c7e0e1cb076af9718"} Nov 01 00:53:29.452609 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.452578 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:53:29.510925 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.510895 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-kube-api-access-lnr67\" (UniqueName: \"kubernetes.io/projected/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-kube-api-access-lnr67\") pod \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\" (UID: \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\") " Nov 01 00:53:29.511122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.511041 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-plugins-config-volume\") pod \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\" (UID: \"0b44ac67-5d41-4d1d-b378-211c5d1b08e1\") " Nov 01 00:53:29.511460 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.511435 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-plugins-config-volume" pod "0b44ac67-5d41-4d1d-b378-211c5d1b08e1" (UID: "0b44ac67-5d41-4d1d-b378-211c5d1b08e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:53:29.511582 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.511567 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:29.513441 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.513416 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-kube-api-access-lnr67" pod "0b44ac67-5d41-4d1d-b378-211c5d1b08e1" (UID: "0b44ac67-5d41-4d1d-b378-211c5d1b08e1"). InnerVolumeSpecName "kube-api-access-lnr67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:29.613069 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:29.613020 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnr67\" (UniqueName: \"kubernetes.io/projected/0b44ac67-5d41-4d1d-b378-211c5d1b08e1-kube-api-access-lnr67\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:30.295823 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:30.295786 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:30.398419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:30.398388 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" Nov 01 00:53:30.398419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:30.398401 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl" event={"ID":"0b44ac67-5d41-4d1d-b378-211c5d1b08e1","Type":"ContainerDied","Data":"f36ac01e77ab749334dcf49264a7c0b2fb736c771df70160ccc77cb0e9029f84"} Nov 01 00:53:30.398667 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:30.398458 6036 scope.go:122] "RemoveContainer" containerID="1fa05f29727f48a3ba19203269c6455cdb37ed6b3b6c637c7e0e1cb076af9718" Nov 01 00:53:30.503321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:30.503273 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:30.568540 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:30.568446 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl"] Nov 01 00:53:30.572200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:30.572176 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a74f100-75d8b55dd8-pjljl"] Nov 01 00:53:30.984191 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:30.984107 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:31.252320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:31.252217 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b44ac67-5d41-4d1d-b378-211c5d1b08e1" path="/var/lib/kubelet/pods/0b44ac67-5d41-4d1d-b378-211c5d1b08e1/volumes" Nov 01 00:53:32.898045 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:32.898009 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:37.439172 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:37.439137 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:42.650069 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:42.650033 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:47.775993 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:47.775958 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:49.507400 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:53:49.507362 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:53:51.994200 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:51.994161 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:53.179852 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:53.179806 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:53.559204 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:53.559163 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:53.864774 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:53.864663 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:54.299535 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:54.299489 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:54.300201 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:54.300100 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f"] Nov 01 00:53:54.300623 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:54.300598 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" containerID="cri-o://98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa" gracePeriod=30 Nov 01 00:53:54.930403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:54.930368 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:55.143298 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:55.143237 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:55.350850 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:55.350805 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:55.896374 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:55.849279 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:56.133089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:56.133057 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:56.352612 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:56.352574 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:56.818739 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:56.818703 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:57.375860 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.375821 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf"] Nov 01 00:53:57.391447 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.391418 6036 cpu_manager.go:414] "containerMap: removing container" podUID="7e804a6c-f36e-46ea-8a04-109911b25c56" containerName="payload-processing" Nov 01 00:53:57.391707 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.391680 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e804a6c-f36e-46ea-8a04-109911b25c56" containerName="payload-processing" Nov 01 00:53:57.391861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.391750 6036 cpu_manager.go:414] "containerMap: removing container" podUID="5d3b391a-7a9d-4bb6-ae51-1efa84db6451" containerName="payload-pre-processing" Nov 01 00:53:57.391861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.391760 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3b391a-7a9d-4bb6-ae51-1efa84db6451" containerName="payload-pre-processing" Nov 01 00:53:57.391861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.391800 6036 cpu_manager.go:414] "containerMap: removing container" podUID="0b44ac67-5d41-4d1d-b378-211c5d1b08e1" containerName="payload-processing" Nov 01 00:53:57.391861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.391808 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b44ac67-5d41-4d1d-b378-211c5d1b08e1" containerName="payload-processing" Nov 01 00:53:57.392078 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.392065 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3b391a-7a9d-4bb6-ae51-1efa84db6451" containerName="payload-pre-processing" Nov 01 00:53:57.392133 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.392084 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b44ac67-5d41-4d1d-b378-211c5d1b08e1" containerName="payload-processing" Nov 01 00:53:57.392133 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.392106 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e804a6c-f36e-46ea-8a04-109911b25c56" containerName="payload-processing" Nov 01 00:53:57.399609 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.399580 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.553668 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.553635 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf"] Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.580430 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-podinfo\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.580505 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-data\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.580556 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-socket\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.580648 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-token\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.580686 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-certs\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.580811 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbbbc8a-1d74-465c-9e10-40840abceb51-istiod-ca-cert\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.580878 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-credential-socket\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.580983 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxhp\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-kube-api-access-nnxhp\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.581318 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.581011 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-envoy\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.584699 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.584672 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5f10727e-openshift-default-dockercfg-5dqjd\"" Nov 01 00:53:57.601904 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.601861 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:57.682867 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.682773 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbbbc8a-1d74-465c-9e10-40840abceb51-istiod-ca-cert\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.682867 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.682836 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-credential-socket\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.683681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.682903 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxhp\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-kube-api-access-nnxhp\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.683681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.682932 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-envoy\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.683681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.683078 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-podinfo\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.683681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.683113 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-data\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.683681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.683156 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-socket\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.683681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.683236 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-token\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.683681 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.683299 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-certs\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.684054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.683752 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-certs\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.684054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.683837 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-data\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.684434 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.684412 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbbbc8a-1d74-465c-9e10-40840abceb51-istiod-ca-cert\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.684630 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.684513 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-credential-socket\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.684630 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.684558 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-socket\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.688095 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.688039 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-podinfo\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.700758 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.700716 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-envoy\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.745948 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.745913 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-token\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.745948 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.746077 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxhp\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-kube-api-access-nnxhp\") pod \"e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.887458 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.887426 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:57.939055 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:57.938945 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:53:58.348702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:58.348665 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf"] Nov 01 00:53:58.385473 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:53:58.385443 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbbbc8a_1d74_465c_9e10_40840abceb51.slice/crio-c2ca76eee9c9e411f06f854d61d5a9d1c3a6fd304717cb9c79cc1d095b7409de WatchSource:0}: Error finding container c2ca76eee9c9e411f06f854d61d5a9d1c3a6fd304717cb9c79cc1d095b7409de: Status 404 returned error can't find the container with id c2ca76eee9c9e411f06f854d61d5a9d1c3a6fd304717cb9c79cc1d095b7409de Nov 01 00:53:58.392529 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:58.392491 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:53:58.392638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:58.392564 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:53:58.392638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:58.392595 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:53:58.945001 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:58.944967 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" event={"ID":"4bbbbc8a-1d74-465c-9e10-40840abceb51","Type":"ContainerStarted","Data":"69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5"} Nov 01 00:53:58.945265 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:58.945019 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" event={"ID":"4bbbbc8a-1d74-465c-9e10-40840abceb51","Type":"ContainerStarted","Data":"c2ca76eee9c9e411f06f854d61d5a9d1c3a6fd304717cb9c79cc1d095b7409de"} Nov 01 00:53:59.120120 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.120066 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podStartSLOduration=2.120046837 podStartE2EDuration="2.120046837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:53:59.117916038 +0000 UTC m=+1783.993137056" watchObservedRunningTime="2026-11-01 00:53:59.120046837 +0000 UTC m=+1783.995267852" Nov 01 00:53:59.621531 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.621500 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:53:59.727400 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727371 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-data\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.727565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727479 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-podinfo\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.727565 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727515 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-token\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-token\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.727639 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727577 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-certs\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.727639 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727614 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-data" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:53:59.727704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727693 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-socket\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.727743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727734 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-credential-socket\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.727816 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727794 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-envoy\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.727974 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727852 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-kube-api-access-6sjfw\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-kube-api-access-6sjfw\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.727974 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727877 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-certs" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:53:59.728098 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727984 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-socket" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:53:59.728098 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.727989 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istiod-ca-cert\") pod \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\" (UID: \"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d\") " Nov 01 00:53:59.728098 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.728025 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-credential-socket" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:53:59.728442 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.728273 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istiod-ca-cert" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:53:59.728833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.728811 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.729005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.728966 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.729005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.728989 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.729005 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.729002 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.729222 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.729017 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.730221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.730197 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-podinfo" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:53:59.730613 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.730585 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-envoy" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:53:59.730734 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.730712 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-kube-api-access-6sjfw" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "kube-api-access-6sjfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:59.730782 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.730766 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-token" pod "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" (UID: "c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:59.830510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.830481 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.830510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.830506 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.830510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.830517 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.830741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.830526 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjfw\" (UniqueName: \"kubernetes.io/projected/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d-kube-api-access-6sjfw\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:59.888534 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.888441 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:59.894044 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.894018 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:59.953705 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.953674 6036 generic.go:356] "Generic (PLEG): container finished" podID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerID="98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa" exitCode=0 Nov 01 00:53:59.953884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.953723 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" Nov 01 00:53:59.953884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.953761 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" event={"ID":"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d","Type":"ContainerDied","Data":"98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa"} Nov 01 00:53:59.953884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.953815 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f" event={"ID":"c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d","Type":"ContainerDied","Data":"4c0bf326d6ec8d3b0faea4c99a60c0a48eff5d2355aa3b6ae55aef1ff5d67d33"} Nov 01 00:53:59.953884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.953839 6036 scope.go:122] "RemoveContainer" containerID="98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa" Nov 01 00:53:59.954107 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.954091 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:59.955116 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.954987 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:53:59.966370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.966352 6036 scope.go:122] "RemoveContainer" containerID="e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676" Nov 01 00:53:59.976367 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.976347 6036 scope.go:122] "RemoveContainer" containerID="98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa" Nov 01 00:53:59.976669 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:53:59.976650 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa\": container with ID starting with 98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa not found: ID does not exist" containerID="98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa" Nov 01 00:53:59.976722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.976679 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa"} err="failed to get container status \"98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa\": rpc error: code = NotFound desc = could not find container \"98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa\": container with ID starting with 98fb77832d5b5d07c2db6d3046f833e93fe9cc7d115a85dabcf36fb8235c02aa not found: ID does not exist" Nov 01 00:53:59.976722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.976698 6036 scope.go:122] "RemoveContainer" containerID="e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676" Nov 01 00:53:59.976941 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:53:59.976921 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676\": container with ID starting with e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676 not found: ID does not exist" containerID="e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676" Nov 01 00:53:59.976979 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:53:59.976948 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676"} err="failed to get container status \"e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676\": rpc error: code = NotFound desc = could not find container \"e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676\": container with ID starting with e9dde92adb30c96ae031f23fda3a7879781684f94c60a26bb47593a47cd41676 not found: ID does not exist" Nov 01 00:54:00.309193 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:00.309158 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:00.327547 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:00.327514 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f"] Nov 01 00:54:00.331849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:00.331825 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-64566096-openshift-default-6bd6ff95b9-2jr7f"] Nov 01 00:54:01.253359 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:01.253327 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" path="/var/lib/kubelet/pods/c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d/volumes" Nov 01 00:54:02.104520 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:02.104488 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:03.528181 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:03.528142 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:03.759576 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:03.759490 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:05.887788 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:05.887742 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:08.495684 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.495607 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:08.715750 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.715704 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:08.868179 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.868140 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg"] Nov 01 00:54:08.869033 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.869010 6036 cpu_manager.go:414] "containerMap: removing container" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" Nov 01 00:54:08.869033 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.869029 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" Nov 01 00:54:08.869226 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.869043 6036 cpu_manager.go:414] "containerMap: removing container" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" Nov 01 00:54:08.869226 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.869048 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" Nov 01 00:54:08.869226 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.869139 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" Nov 01 00:54:08.873346 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.873325 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:08.922034 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.921990 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4158ce-fc83-4aaa-a33e-393e67ae0559-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg\" (UID: \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:08.922502 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.922478 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64p8\" (UniqueName: \"kubernetes.io/projected/7b4158ce-fc83-4aaa-a33e-393e67ae0559-kube-api-access-m64p8\") pod \"payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg\" (UID: \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:08.944645 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.944556 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg"] Nov 01 00:54:08.987033 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.986994 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5f10727e-dockercfg-8bktm\"" Nov 01 00:54:08.987237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:08.987110 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5f10727e\"" Nov 01 00:54:09.024581 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:09.024546 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4158ce-fc83-4aaa-a33e-393e67ae0559-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg\" (UID: \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:09.024788 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:09.024669 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64p8\" (UniqueName: \"kubernetes.io/projected/7b4158ce-fc83-4aaa-a33e-393e67ae0559-kube-api-access-m64p8\") pod \"payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg\" (UID: \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:09.025725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:09.025698 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4158ce-fc83-4aaa-a33e-393e67ae0559-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg\" (UID: \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:09.092148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:09.088930 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64p8\" (UniqueName: \"kubernetes.io/projected/7b4158ce-fc83-4aaa-a33e-393e67ae0559-kube-api-access-m64p8\") pod \"payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg\" (UID: \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:09.196729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:09.196623 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:09.656111 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:09.656066 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg"] Nov 01 00:54:09.656540 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:54:09.656086 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4158ce_fc83_4aaa_a33e_393e67ae0559.slice/crio-c021cb459ccaff092f983d9cd57321886fc39649070ffa2d6bca3f69347cb6a4 WatchSource:0}: Error finding container c021cb459ccaff092f983d9cd57321886fc39649070ffa2d6bca3f69347cb6a4: Status 404 returned error can't find the container with id c021cb459ccaff092f983d9cd57321886fc39649070ffa2d6bca3f69347cb6a4 Nov 01 00:54:10.036493 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:10.036459 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" event={"ID":"7b4158ce-fc83-4aaa-a33e-393e67ae0559","Type":"ContainerStarted","Data":"77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b"} Nov 01 00:54:10.036688 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:10.036522 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" event={"ID":"7b4158ce-fc83-4aaa-a33e-393e67ae0559","Type":"ContainerStarted","Data":"c021cb459ccaff092f983d9cd57321886fc39649070ffa2d6bca3f69347cb6a4"} Nov 01 00:54:10.036688 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:10.036545 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:10.157011 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:10.156956 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" podStartSLOduration=2.156942085 podStartE2EDuration="2.156942085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:54:10.155376818 +0000 UTC m=+1795.030597832" watchObservedRunningTime="2026-11-01 00:54:10.156942085 +0000 UTC m=+1795.032163100" Nov 01 00:54:11.056178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:11.056138 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:13.528758 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:13.528723 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:13.811147 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:13.811115 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:14.399188 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:14.399147 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:16.370213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:16.370182 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:16.699931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:16.699843 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:19.506641 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:54:19.506604 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:54:20.732043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:20.732010 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6"] Nov 01 00:54:20.732477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:20.732222 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" podUID="0ca7d99e-855e-4290-8e64-5efe096b5c0d" containerName="maas-api" containerID="cri-o://878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3" gracePeriod=30 Nov 01 00:54:20.931264 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:20.931174 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:21.064539 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.064510 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:54:21.076491 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.076458 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:54:21.161312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.149261 6036 generic.go:356] "Generic (PLEG): container finished" podID="0ca7d99e-855e-4290-8e64-5efe096b5c0d" containerID="878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3" exitCode=0 Nov 01 00:54:21.161312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.149656 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" event={"ID":"0ca7d99e-855e-4290-8e64-5efe096b5c0d","Type":"ContainerDied","Data":"878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3"} Nov 01 00:54:21.161312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.149710 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" event={"ID":"0ca7d99e-855e-4290-8e64-5efe096b5c0d","Type":"ContainerDied","Data":"288630052aae97fdce8a7ea755964fcadcc2c0f8b45485dac7c3e5c3239e1093"} Nov 01 00:54:21.161312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.149765 6036 scope.go:122] "RemoveContainer" containerID="878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3" Nov 01 00:54:21.161312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.149992 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6" Nov 01 00:54:21.174917 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.174891 6036 scope.go:122] "RemoveContainer" containerID="878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3" Nov 01 00:54:21.175443 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:54:21.175411 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3\": container with ID starting with 878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3 not found: ID does not exist" containerID="878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3" Nov 01 00:54:21.175578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.175454 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3"} err="failed to get container status \"878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3\": rpc error: code = NotFound desc = could not find container \"878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3\": container with ID starting with 878ea131301150cc40240261caa77e12219d6a6ca956da67725f5e7e0e1dd6b3 not found: ID does not exist" Nov 01 00:54:21.177303 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.177259 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca7d99e-855e-4290-8e64-5efe096b5c0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca7d99e-855e-4290-8e64-5efe096b5c0d-openshift-service-ca\") pod \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " Nov 01 00:54:21.177430 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.177381 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-metrics-certs\") pod \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " Nov 01 00:54:21.177547 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.177528 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls\") pod \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " Nov 01 00:54:21.177617 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.177574 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca7d99e-855e-4290-8e64-5efe096b5c0d-openshift-service-ca" pod "0ca7d99e-855e-4290-8e64-5efe096b5c0d" (UID: "0ca7d99e-855e-4290-8e64-5efe096b5c0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:54:21.177880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.177859 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca7d99e-855e-4290-8e64-5efe096b5c0d-kube-api-access-2ffs4\" (UniqueName: \"kubernetes.io/projected/0ca7d99e-855e-4290-8e64-5efe096b5c0d-kube-api-access-2ffs4\") pod \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\" (UID: \"0ca7d99e-855e-4290-8e64-5efe096b5c0d\") " Nov 01 00:54:21.178780 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.178756 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca7d99e-855e-4290-8e64-5efe096b5c0d-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:21.179888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.179863 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls" pod "0ca7d99e-855e-4290-8e64-5efe096b5c0d" (UID: "0ca7d99e-855e-4290-8e64-5efe096b5c0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:54:21.179888 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.179880 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-metrics-certs" pod "0ca7d99e-855e-4290-8e64-5efe096b5c0d" (UID: "0ca7d99e-855e-4290-8e64-5efe096b5c0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:54:21.180403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.180381 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca7d99e-855e-4290-8e64-5efe096b5c0d-kube-api-access-2ffs4" pod "0ca7d99e-855e-4290-8e64-5efe096b5c0d" (UID: "0ca7d99e-855e-4290-8e64-5efe096b5c0d"). InnerVolumeSpecName "kube-api-access-2ffs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:54:21.280778 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.280747 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffs4\" (UniqueName: \"kubernetes.io/projected/0ca7d99e-855e-4290-8e64-5efe096b5c0d-kube-api-access-2ffs4\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:21.281013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.280987 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:21.281013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.281014 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca7d99e-855e-4290-8e64-5efe096b5c0d-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:21.571096 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.571061 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6"] Nov 01 00:54:21.574365 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:21.574342 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-14957a74-5b4d4d56bb-7sqv6"] Nov 01 00:54:22.521777 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:22.521745 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:22.832875 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:22.832827 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:23.099674 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:23.099589 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:23.292369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:23.290022 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca7d99e-855e-4290-8e64-5efe096b5c0d" path="/var/lib/kubelet/pods/0ca7d99e-855e-4290-8e64-5efe096b5c0d/volumes" Nov 01 00:54:23.338312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:23.334663 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:24.157070 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:24.156998 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:24.413721 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:24.413636 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:24.657857 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:24.657825 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:28.430931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:28.430889 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:30.706216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:30.706176 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:31.376804 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:31.376770 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:31.596909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:31.596861 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:33.440884 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:33.440851 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:34.913325 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.913275 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj"] Nov 01 00:54:34.914239 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.914215 6036 cpu_manager.go:414] "containerMap: removing container" podUID="0ca7d99e-855e-4290-8e64-5efe096b5c0d" containerName="maas-api" Nov 01 00:54:34.914370 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.914239 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca7d99e-855e-4290-8e64-5efe096b5c0d" containerName="maas-api" Nov 01 00:54:34.914510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.914496 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca7d99e-855e-4290-8e64-5efe096b5c0d" containerName="maas-api" Nov 01 00:54:34.914573 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.914518 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8880c8e-a9f3-41b3-87c6-cf1b8b49d47d" containerName="istio-proxy" Nov 01 00:54:34.918452 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.918431 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:34.972722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.972681 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 00:54:34.972952 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.972749 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 00:54:34.972952 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:34.972791 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 00:54:35.023673 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.023640 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lqx6r\"" Nov 01 00:54:35.025924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.025887 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf7dp\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-kube-api-access-vf7dp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.026097 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.026076 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.026197 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.026177 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.026266 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.026209 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.026440 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.026412 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.026512 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.026451 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c228fd-2e0f-4c82-a177-1bae31193c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.026512 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.026491 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.027849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.027824 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj"] Nov 01 00:54:35.127757 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.127725 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.127971 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.127768 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c228fd-2e0f-4c82-a177-1bae31193c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.127971 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.127794 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.127971 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.127896 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf7dp\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-kube-api-access-vf7dp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.128154 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.127989 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.128154 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.128066 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.128154 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.128092 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.128853 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.128822 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.130333 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.130304 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c228fd-2e0f-4c82-a177-1bae31193c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.130453 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.130432 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.130783 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.130756 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.131127 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.131108 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.192047 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.191970 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.197213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.197180 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf7dp\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-kube-api-access-vf7dp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k98tj\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.231471 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.231442 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:35.680213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:35.680187 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj"] Nov 01 00:54:35.680528 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:54:35.680504 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c228fd_2e0f_4c82_a177_1bae31193c6a.slice/crio-d750ba4d51a3eb007bd0c88feba2b5fa5f2cb7cf079463c91490fcfbf282c78d WatchSource:0}: Error finding container d750ba4d51a3eb007bd0c88feba2b5fa5f2cb7cf079463c91490fcfbf282c78d: Status 404 returned error can't find the container with id d750ba4d51a3eb007bd0c88feba2b5fa5f2cb7cf079463c91490fcfbf282c78d Nov 01 00:54:36.263424 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:36.263386 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" event={"ID":"b0c228fd-2e0f-4c82-a177-1bae31193c6a","Type":"ContainerStarted","Data":"d750ba4d51a3eb007bd0c88feba2b5fa5f2cb7cf079463c91490fcfbf282c78d"} Nov 01 00:54:38.338451 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:38.338387 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:39.325982 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:39.325946 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:54:39.326095 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:39.326023 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:54:40.299081 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:40.299044 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" event={"ID":"b0c228fd-2e0f-4c82-a177-1bae31193c6a","Type":"ContainerStarted","Data":"8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5"} Nov 01 00:54:40.299540 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:40.299314 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:40.301008 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:40.300987 6036 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-k98tj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:54:40.301129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:40.301029 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" podUID="b0c228fd-2e0f-4c82-a177-1bae31193c6a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:54:40.512204 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:40.512153 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" podStartSLOduration=2.870746263 podStartE2EDuration="6.512137226s" totalImagesPullingTime="3.641390963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:54:40.511061003 +0000 UTC m=+1825.386282018" watchObservedRunningTime="2026-11-01 00:54:40.512137226 +0000 UTC m=+1825.387358241" Nov 01 00:54:40.723773 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:40.723663 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:41.306188 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:41.306160 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 00:54:43.218641 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:43.218601 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:43.455511 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:43.455474 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:43.679865 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:43.679832 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:46.632198 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:46.632164 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:48.179364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:48.179325 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:48.394271 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:48.394235 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:49.064038 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:49.063991 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:49.273557 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:49.273520 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:49.507695 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:54:49.507602 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:54:53.235679 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:53.235624 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:55.229931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:55.229898 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:55.904217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:55.904171 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:56.131703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:56.131645 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:54:58.122416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:54:58.122368 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:00.818835 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:00.818797 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:01.036619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:01.036585 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:03.251758 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:03.251705 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:04.048187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:04.048151 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:04.262976 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:04.262939 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:08.342821 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:08.342781 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:13.163696 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:13.163654 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:15.995480 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:15.995435 6036 patch_prober.go:29] interesting pod/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:55:15.995946 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:15.995516 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:55:17.679360 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:17.679326 6036 generic.go:356] "Generic (PLEG): container finished" podID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerID="69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5" exitCode=0 Nov 01 00:55:17.679926 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:17.679396 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" event={"ID":"4bbbbc8a-1d74-465c-9e10-40840abceb51","Type":"ContainerDied","Data":"69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5"} Nov 01 00:55:17.679926 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:17.679909 6036 scope.go:122] "RemoveContainer" containerID="69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5" Nov 01 00:55:17.681482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:17.681451 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:55:17.681593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:17.681517 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:55:17.681593 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:17.681548 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:55:18.089216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:18.089166 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:18.688637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:18.688600 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" event={"ID":"4bbbbc8a-1d74-465c-9e10-40840abceb51","Type":"ContainerStarted","Data":"68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e"} Nov 01 00:55:18.887896 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:18.887861 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:55:18.888081 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:18.888027 6036 patch_prober.go:29] interesting pod/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 00:55:18.888127 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:18.888082 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 00:55:19.508237 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:55:19.508203 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:55:19.888581 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:19.888547 6036 patch_prober.go:29] interesting pod/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 00:55:19.889072 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:19.888617 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 00:55:20.888856 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:20.888814 6036 patch_prober.go:29] interesting pod/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 00:55:20.889319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:20.888877 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 00:55:21.888541 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:21.888498 6036 patch_prober.go:29] interesting pod/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 00:55:21.888739 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:21.888559 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 00:55:22.888439 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:22.888407 6036 patch_prober.go:29] interesting pod/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 00:55:22.889214 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:22.888460 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 00:55:22.941983 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:22.941944 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:23.887835 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:23.887803 6036 patch_prober.go:29] interesting pod/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 00:55:23.888025 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:23.887857 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 00:55:24.892950 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:24.892920 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:55:24.893413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:24.893328 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:55:24.894096 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:24.894078 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:55:25.610714 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:25.610678 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:27.858974 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:27.858936 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:31.093788 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:31.093742 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:32.728501 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:32.728429 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:33.158418 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:33.158350 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:33.800195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:33.800158 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:34.024907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:34.024872 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:38.060013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:38.059968 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:38.330764 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:38.330676 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:38.555506 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:38.555446 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:43.242873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:43.242835 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:43.465873 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:43.465838 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:43.700836 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:43.700757 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:48.500335 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:48.498112 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:49.508117 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:55:49.508085 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:55:53.683859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:53.683817 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:53.988841 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:53.988750 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:54.211849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:54.211764 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:56.664674 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:56.664633 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:56.883697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:56.883648 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:55:58.830841 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:55:58.827950 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:02.965278 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:02.965230 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:03.547901 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:03.547859 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg"] Nov 01 00:56:03.664049 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:03.663766 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:03.923483 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:03.922168 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:04.131461 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:04.131409 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:06.640423 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:06.640385 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:08.522992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:08.522957 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:08.549349 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:08.549255 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" podUID="7b4158ce-fc83-4aaa-a33e-393e67ae0559" containerName="payload-pre-processing" containerID="cri-o://77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b" gracePeriod=25 Nov 01 00:56:08.856210 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:08.856176 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:09.023637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.023597 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:56:09.107777 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.107676 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:09.109100 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.109074 6036 generic.go:356] "Generic (PLEG): container finished" podID="7b4158ce-fc83-4aaa-a33e-393e67ae0559" containerID="77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b" exitCode=0 Nov 01 00:56:09.109241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.109130 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" event={"ID":"7b4158ce-fc83-4aaa-a33e-393e67ae0559","Type":"ContainerDied","Data":"77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b"} Nov 01 00:56:09.109241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.109139 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" Nov 01 00:56:09.109241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.109204 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg" event={"ID":"7b4158ce-fc83-4aaa-a33e-393e67ae0559","Type":"ContainerDied","Data":"c021cb459ccaff092f983d9cd57321886fc39649070ffa2d6bca3f69347cb6a4"} Nov 01 00:56:09.109241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.109231 6036 scope.go:122] "RemoveContainer" containerID="77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b" Nov 01 00:56:09.120703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.120680 6036 scope.go:122] "RemoveContainer" containerID="77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b" Nov 01 00:56:09.121109 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:56:09.121065 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b\": container with ID starting with 77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b not found: ID does not exist" containerID="77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b" Nov 01 00:56:09.121227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.121122 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b"} err="failed to get container status \"77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b\": rpc error: code = NotFound desc = could not find container \"77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b\": container with ID starting with 77055da88eccb24d521e114f511a7a3d13e398d2b6b188d1ca94c18e99bf540b not found: ID does not exist" Nov 01 00:56:09.161608 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.161576 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4158ce-fc83-4aaa-a33e-393e67ae0559-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4158ce-fc83-4aaa-a33e-393e67ae0559-plugins-config-volume\") pod \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\" (UID: \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\") " Nov 01 00:56:09.161791 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.161719 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4158ce-fc83-4aaa-a33e-393e67ae0559-kube-api-access-m64p8\" (UniqueName: \"kubernetes.io/projected/7b4158ce-fc83-4aaa-a33e-393e67ae0559-kube-api-access-m64p8\") pod \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\" (UID: \"7b4158ce-fc83-4aaa-a33e-393e67ae0559\") " Nov 01 00:56:09.162068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.162038 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4158ce-fc83-4aaa-a33e-393e67ae0559-plugins-config-volume" pod "7b4158ce-fc83-4aaa-a33e-393e67ae0559" (UID: "7b4158ce-fc83-4aaa-a33e-393e67ae0559"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:09.162798 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.162767 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4158ce-fc83-4aaa-a33e-393e67ae0559-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:09.164227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.164200 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4158ce-fc83-4aaa-a33e-393e67ae0559-kube-api-access-m64p8" pod "7b4158ce-fc83-4aaa-a33e-393e67ae0559" (UID: "7b4158ce-fc83-4aaa-a33e-393e67ae0559"). InnerVolumeSpecName "kube-api-access-m64p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:09.264708 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.264681 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m64p8\" (UniqueName: \"kubernetes.io/projected/7b4158ce-fc83-4aaa-a33e-393e67ae0559-kube-api-access-m64p8\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:09.532217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.532138 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg"] Nov 01 00:56:09.535557 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:09.535535 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f10727e-7c57b44ff-bjgdg"] Nov 01 00:56:11.252301 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:11.252244 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4158ce-fc83-4aaa-a33e-393e67ae0559" path="/var/lib/kubelet/pods/7b4158ce-fc83-4aaa-a33e-393e67ae0559/volumes" Nov 01 00:56:11.790015 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:11.789980 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:12.055668 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:12.055574 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:13.418243 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:13.418204 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:13.651220 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:13.651186 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:13.870997 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:13.870960 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:14.097018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:14.096983 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:16.813303 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:16.813253 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:17.026909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:17.026855 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:18.918657 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:18.918616 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:19.510051 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:56:19.510013 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:56:23.981638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:23.981599 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:25.544670 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:25.544635 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw"] Nov 01 00:56:25.545089 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:25.544859 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" containerID="cri-o://03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a" gracePeriod=30 Nov 01 00:56:25.595910 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:25.595874 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2"] Nov 01 00:56:25.596156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:25.596117 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" containerID="cri-o://eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a" gracePeriod=30 Nov 01 00:56:28.255697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:28.255661 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:30.021533 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:30.021492 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:30.249793 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:30.249706 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:30.460894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:30.460861 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:30.624148 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:30.624114 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 00:56:30.624346 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:30.624184 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 00:56:30.877145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:30.876928 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:30.912716 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:30.912686 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:56:30.941586 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:30.941560 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:56:31.007493 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007404 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-credential-socket\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.007493 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007448 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-certs\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.007685 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007559 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-podinfo\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.007685 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007621 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-podinfo\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.007685 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007651 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-kube-api-access-jxrbk\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-kube-api-access-jxrbk\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.007818 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007713 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-credential-socket\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.007818 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007710 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-credential-socket" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.007818 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007795 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-certs" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.008003 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007828 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-envoy\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.008003 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007856 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-data\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.008003 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007917 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-envoy\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.008003 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.007959 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-socket\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.008221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008031 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-certs\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.008221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008042 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-credential-socket" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.008221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008099 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-socket\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.008221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008134 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-data\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.008221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008197 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istiod-ca-cert\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.008221 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008195 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-data" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.008672 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008227 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-kube-api-access-jp8xv\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-kube-api-access-jp8xv\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.008672 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008256 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-token\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-token\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.008672 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008309 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-token\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-token\") pod \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\" (UID: \"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57\") " Nov 01 00:56:31.008672 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008351 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e14247-ec2a-481c-a38a-e4030afb6257-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e14247-ec2a-481c-a38a-e4030afb6257-istiod-ca-cert\") pod \"a4e14247-ec2a-481c-a38a-e4030afb6257\" (UID: \"a4e14247-ec2a-481c-a38a-e4030afb6257\") " Nov 01 00:56:31.008672 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008641 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-socket" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.009103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.008686 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-data" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.009103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009024 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istiod-ca-cert" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:31.009103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009030 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-socket" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.009486 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009460 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.009486 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009474 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e14247-ec2a-481c-a38a-e4030afb6257-istiod-ca-cert" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:31.009629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009491 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.009629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009506 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.009629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009520 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.009629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009534 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.009629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009547 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.009629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009560 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.009629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009574 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.009981 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.009643 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-certs" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.011203 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.011173 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-envoy" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.011336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.011200 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-podinfo" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:56:31.011336 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.011217 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-podinfo" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:56:31.011476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.011366 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-token" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:31.011539 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.011487 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-kube-api-access-jxrbk" pod "a4e14247-ec2a-481c-a38a-e4030afb6257" (UID: "a4e14247-ec2a-481c-a38a-e4030afb6257"). InnerVolumeSpecName "kube-api-access-jxrbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:31.011743 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.011723 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-kube-api-access-jp8xv" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "kube-api-access-jp8xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:31.011978 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.011962 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-envoy" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:31.012102 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.012085 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-token" pod "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" (UID: "1e5a3620-fb09-4cb9-95e2-ddafe9e12a57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:31.111521 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111468 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.111521 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111505 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.111521 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111519 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e14247-ec2a-481c-a38a-e4030afb6257-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.112145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111532 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8xv\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-kube-api-access-jp8xv\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.112145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111546 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.112145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111557 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.112145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111568 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e14247-ec2a-481c-a38a-e4030afb6257-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.112145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111581 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.112145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111594 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e14247-ec2a-481c-a38a-e4030afb6257-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.112145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.111606 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxrbk\" (UniqueName: \"kubernetes.io/projected/a4e14247-ec2a-481c-a38a-e4030afb6257-kube-api-access-jxrbk\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.225316 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.224210 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:31.290210 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.290174 6036 generic.go:356] "Generic (PLEG): container finished" podID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerID="eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a" exitCode=0 Nov 01 00:56:31.290412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.290230 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" Nov 01 00:56:31.290412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.290258 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" event={"ID":"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57","Type":"ContainerDied","Data":"eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a"} Nov 01 00:56:31.290412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.290367 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2" event={"ID":"1e5a3620-fb09-4cb9-95e2-ddafe9e12a57","Type":"ContainerDied","Data":"0cb4eb51984c1f17a34b6cc68e826bafd5ebf03c6a1f142f150c0de12bb8ba6a"} Nov 01 00:56:31.290412 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.290399 6036 scope.go:122] "RemoveContainer" containerID="eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a" Nov 01 00:56:31.292422 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.292386 6036 generic.go:356] "Generic (PLEG): container finished" podID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerID="03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a" exitCode=0 Nov 01 00:56:31.292549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.292424 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" event={"ID":"a4e14247-ec2a-481c-a38a-e4030afb6257","Type":"ContainerDied","Data":"03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a"} Nov 01 00:56:31.292549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.292464 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" Nov 01 00:56:31.292549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.292479 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw" event={"ID":"a4e14247-ec2a-481c-a38a-e4030afb6257","Type":"ContainerDied","Data":"dfafa80847a3c9a2d84f5169073602a7b2f68f26b5a1b3828107d7eeac4647a7"} Nov 01 00:56:31.307464 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.307438 6036 scope.go:122] "RemoveContainer" containerID="eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a" Nov 01 00:56:31.307881 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:56:31.307847 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a\": container with ID starting with eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a not found: ID does not exist" containerID="eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a" Nov 01 00:56:31.308012 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.307895 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a"} err="failed to get container status \"eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a\": rpc error: code = NotFound desc = could not find container \"eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a\": container with ID starting with eac8c8baf1e9bb2af96a76d810c77f5c6253d11aabab2b333acfea60ae52176a not found: ID does not exist" Nov 01 00:56:31.308012 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.307924 6036 scope.go:122] "RemoveContainer" containerID="03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a" Nov 01 00:56:31.322514 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.322489 6036 scope.go:122] "RemoveContainer" containerID="03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a" Nov 01 00:56:31.322926 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:56:31.322900 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a\": container with ID starting with 03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a not found: ID does not exist" containerID="03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a" Nov 01 00:56:31.323053 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.322932 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a"} err="failed to get container status \"03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a\": rpc error: code = NotFound desc = could not find container \"03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a\": container with ID starting with 03cf61078d2dad263a2516e22e1ef8cdda55c835da30ed2c54f056e309a1a70a not found: ID does not exist" Nov 01 00:56:31.439654 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.439618 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:31.463543 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.463510 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw"] Nov 01 00:56:31.466935 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.466913 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-tdcnw"] Nov 01 00:56:31.583334 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.582006 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2"] Nov 01 00:56:31.587645 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:31.587620 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-48ks2"] Nov 01 00:56:33.048137 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:33.048095 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:33.254862 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:33.254828 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" path="/var/lib/kubelet/pods/1e5a3620-fb09-4cb9-95e2-ddafe9e12a57/volumes" Nov 01 00:56:33.255561 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:33.255543 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" path="/var/lib/kubelet/pods/a4e14247-ec2a-481c-a38a-e4030afb6257/volumes" Nov 01 00:56:33.859782 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:33.859754 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.417410 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn"] Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418543 6036 cpu_manager.go:414] "containerMap: removing container" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418563 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418587 6036 cpu_manager.go:414] "containerMap: removing container" podUID="7b4158ce-fc83-4aaa-a33e-393e67ae0559" containerName="payload-pre-processing" Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418596 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4158ce-fc83-4aaa-a33e-393e67ae0559" containerName="payload-pre-processing" Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418669 6036 cpu_manager.go:414] "containerMap: removing container" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418677 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418822 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e14247-ec2a-481c-a38a-e4030afb6257" containerName="istio-proxy" Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418834 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4158ce-fc83-4aaa-a33e-393e67ae0559" containerName="payload-pre-processing" Nov 01 00:56:34.420320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.418856 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5a3620-fb09-4cb9-95e2-ddafe9e12a57" containerName="istio-proxy" Nov 01 00:56:34.449819 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.426020 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.489889 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.489837 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-27dee6-openshift-default-dockercfg-p6f96\"" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.505633 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-podinfo\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.505680 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-credential-socket\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.505700 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-envoy\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.505767 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-token\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.505795 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-socket\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.505851 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istiod-ca-cert\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.505876 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-certs\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.505940 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpq96\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-kube-api-access-cpq96\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.508722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.506048 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-data\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.551346 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.551128 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn"] Nov 01 00:56:34.609310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.609187 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-token\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.609310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.609260 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-socket\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.609602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.609382 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istiod-ca-cert\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.609602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.609426 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-certs\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.609602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.609527 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpq96\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-kube-api-access-cpq96\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.610082 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.610053 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-socket\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.610279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.610258 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-data\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.610279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.610348 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-podinfo\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.610489 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.610404 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-credential-socket\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.610489 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.610432 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-envoy\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.611122 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.611096 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-certs\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.612364 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.612276 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istiod-ca-cert\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.612901 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.612876 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-credential-socket\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.612996 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.612968 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-data\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.615048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.614932 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:34.619957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.619748 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-podinfo\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.619957 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.619909 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-envoy\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.675387 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.675312 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-token\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.676669 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.676649 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpq96\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-kube-api-access-cpq96\") pod \"e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:34.812083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:34.812045 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:35.082264 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:35.082219 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:35.265658 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:35.265632 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn"] Nov 01 00:56:35.265765 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:56:35.265693 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c93b96_6c9c_4cbc_901a_dc98de25ec1a.slice/crio-7a2a062723b193fddc58ea4e7030f7238c4f0e232d8c3e6414a466f8e13a490d WatchSource:0}: Error finding container 7a2a062723b193fddc58ea4e7030f7238c4f0e232d8c3e6414a466f8e13a490d: Status 404 returned error can't find the container with id 7a2a062723b193fddc58ea4e7030f7238c4f0e232d8c3e6414a466f8e13a490d Nov 01 00:56:35.268470 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:35.268437 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:35.268534 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:35.268506 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:35.268573 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:35.268538 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:35.335164 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:35.333372 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" event={"ID":"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a","Type":"ContainerStarted","Data":"7a2a062723b193fddc58ea4e7030f7238c4f0e232d8c3e6414a466f8e13a490d"} Nov 01 00:56:35.434535 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:35.434490 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:36.421124 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:36.421070 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" event={"ID":"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a","Type":"ContainerStarted","Data":"dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8"} Nov 01 00:56:36.555550 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:36.555488 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" podStartSLOduration=2.55546614 podStartE2EDuration="2.55546614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:36.54890066 +0000 UTC m=+1941.424121674" watchObservedRunningTime="2026-11-01 00:56:36.55546614 +0000 UTC m=+1941.430687156" Nov 01 00:56:36.823940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:36.823912 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:36.863310 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:36.862804 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:37.243378 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:37.243249 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:37.428046 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:37.428013 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:37.428972 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:37.428949 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:56:37.523767 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:37.523681 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:37.760582 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:37.744967 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:37.761072 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:37.761003 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf"] Nov 01 00:56:37.761359 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:37.761308 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" containerID="cri-o://68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e" gracePeriod=30 Nov 01 00:56:37.960373 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:37.957983 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:38.673644 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:38.673592 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:39.669545 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:39.669508 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:39.893614 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:39.893581 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:39.895483 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:39.895454 6036 patch_prober.go:29] interesting pod/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:56:39.895624 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:39.895521 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:56:40.161319 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:40.161267 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:43.075348 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.068206 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:43.179006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.178947 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.260558 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-socket\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.260640 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-podinfo\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.260689 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-kube-api-access-nnxhp\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-kube-api-access-nnxhp\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.260734 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-certs\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.260805 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-data\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.260965 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-token\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-token\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.261004 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-envoy\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.261090 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bbbbc8a-1d74-465c-9e10-40840abceb51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbbbc8a-1d74-465c-9e10-40840abceb51-istiod-ca-cert\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.261132 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-credential-socket\") pod \"4bbbbc8a-1d74-465c-9e10-40840abceb51\" (UID: \"4bbbbc8a-1d74-465c-9e10-40840abceb51\") " Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.262516 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-credential-socket" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.262911 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-socket" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:43.263934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.263838 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-data" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:43.265694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.265668 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbbbc8a-1d74-465c-9e10-40840abceb51-istiod-ca-cert" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:43.265927 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.265906 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-certs" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:43.269622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.269591 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-envoy" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:43.270343 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.270309 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-token" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:43.270844 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.270812 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-kube-api-access-nnxhp" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "kube-api-access-nnxhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:43.271157 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.271138 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-podinfo" pod "4bbbbc8a-1d74-465c-9e10-40840abceb51" (UID: "4bbbbc8a-1d74-465c-9e10-40840abceb51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:56:43.276534 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.276509 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364539 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364561 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364571 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbbbc8a-1d74-465c-9e10-40840abceb51-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364588 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364601 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364611 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364619 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxhp\" (UniqueName: \"kubernetes.io/projected/4bbbbc8a-1d74-465c-9e10-40840abceb51-kube-api-access-nnxhp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364629 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.366404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.364642 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbbbc8a-1d74-465c-9e10-40840abceb51-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:43.481229 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.481189 6036 generic.go:356] "Generic (PLEG): container finished" podID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerID="68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e" exitCode=0 Nov 01 00:56:43.481475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.481275 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" Nov 01 00:56:43.481475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.481465 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" event={"ID":"4bbbbc8a-1d74-465c-9e10-40840abceb51","Type":"ContainerDied","Data":"68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e"} Nov 01 00:56:43.481602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.481521 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf" event={"ID":"4bbbbc8a-1d74-465c-9e10-40840abceb51","Type":"ContainerDied","Data":"c2ca76eee9c9e411f06f854d61d5a9d1c3a6fd304717cb9c79cc1d095b7409de"} Nov 01 00:56:43.481602 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.481553 6036 scope.go:122] "RemoveContainer" containerID="68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e" Nov 01 00:56:43.513578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.513554 6036 scope.go:122] "RemoveContainer" containerID="69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5" Nov 01 00:56:43.534280 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.534257 6036 scope.go:122] "RemoveContainer" containerID="68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e" Nov 01 00:56:43.541405 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:56:43.541365 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e\": container with ID starting with 68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e not found: ID does not exist" containerID="68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e" Nov 01 00:56:43.541584 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.541412 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e"} err="failed to get container status \"68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e\": rpc error: code = NotFound desc = could not find container \"68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e\": container with ID starting with 68934ec67800889863215f25099dff10dcf4238456432335df0400d06430f47e not found: ID does not exist" Nov 01 00:56:43.541584 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.541437 6036 scope.go:122] "RemoveContainer" containerID="69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5" Nov 01 00:56:43.542071 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:56:43.542004 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5\": container with ID starting with 69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5 not found: ID does not exist" containerID="69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5" Nov 01 00:56:43.542192 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.542069 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5"} err="failed to get container status \"69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5\": rpc error: code = NotFound desc = could not find container \"69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5\": container with ID starting with 69f855b58a5f9f486132ae36d7360a1feebf7c1843bded2687b373eab79320e5 not found: ID does not exist" Nov 01 00:56:43.668112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.667978 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf"] Nov 01 00:56:43.674216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:43.674176 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5f10727e-openshift-default-7f48dfc789-jnwcf"] Nov 01 00:56:44.365068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:44.365036 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:44.613477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:44.613448 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:45.263726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:45.263695 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" path="/var/lib/kubelet/pods/4bbbbc8a-1d74-465c-9e10-40840abceb51/volumes" Nov 01 00:56:47.918759 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:47.918729 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:48.129568 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.129524 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:48.284552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.284525 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9"] Nov 01 00:56:48.285225 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.285210 6036 cpu_manager.go:414] "containerMap: removing container" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" Nov 01 00:56:48.285277 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.285227 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" Nov 01 00:56:48.285277 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.285254 6036 cpu_manager.go:414] "containerMap: removing container" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" Nov 01 00:56:48.285277 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.285259 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" Nov 01 00:56:48.285419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.285389 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" Nov 01 00:56:48.290110 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.290091 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:48.347082 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.347046 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9"] Nov 01 00:56:48.347327 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.347176 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:48.362646 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.362600 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a400a2-938a-4ff3-9c1e-5c0f51baf810-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9\" (UID: \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:48.362846 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.362740 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plh4b\" (UniqueName: \"kubernetes.io/projected/82a400a2-938a-4ff3-9c1e-5c0f51baf810-kube-api-access-plh4b\") pod \"payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9\" (UID: \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:48.395354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.393360 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-af82f1-dockercfg-kf5wm\"" Nov 01 00:56:48.395354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.394275 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-af82f1\"" Nov 01 00:56:48.464920 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.464881 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a400a2-938a-4ff3-9c1e-5c0f51baf810-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9\" (UID: \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:48.465396 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.465366 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plh4b\" (UniqueName: \"kubernetes.io/projected/82a400a2-938a-4ff3-9c1e-5c0f51baf810-kube-api-access-plh4b\") pod \"payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9\" (UID: \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:48.465902 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.465876 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a400a2-938a-4ff3-9c1e-5c0f51baf810-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9\" (UID: \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:48.528206 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.528170 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plh4b\" (UniqueName: \"kubernetes.io/projected/82a400a2-938a-4ff3-9c1e-5c0f51baf810-kube-api-access-plh4b\") pod \"payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9\" (UID: \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:48.627360 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:48.626668 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:49.072430 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:49.072400 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9"] Nov 01 00:56:49.073511 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:56:49.073481 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a400a2_938a_4ff3_9c1e_5c0f51baf810.slice/crio-28555382b9d5ab8cd968d98556695dc8ddcf4f191b8822810425d22f84de092a WatchSource:0}: Error finding container 28555382b9d5ab8cd968d98556695dc8ddcf4f191b8822810425d22f84de092a: Status 404 returned error can't find the container with id 28555382b9d5ab8cd968d98556695dc8ddcf4f191b8822810425d22f84de092a Nov 01 00:56:49.507385 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:56:49.507260 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:56:49.531499 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:49.531462 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" event={"ID":"82a400a2-938a-4ff3-9c1e-5c0f51baf810","Type":"ContainerStarted","Data":"3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1"} Nov 01 00:56:49.531666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:49.531510 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" event={"ID":"82a400a2-938a-4ff3-9c1e-5c0f51baf810","Type":"ContainerStarted","Data":"28555382b9d5ab8cd968d98556695dc8ddcf4f191b8822810425d22f84de092a"} Nov 01 00:56:49.531666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:49.531656 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:56:49.651463 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:49.651407 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" podStartSLOduration=1.6513924530000001 podStartE2EDuration="1.651392453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:49.649075895 +0000 UTC m=+1954.524296921" watchObservedRunningTime="2026-11-01 00:56:49.651392453 +0000 UTC m=+1954.526613467" Nov 01 00:56:53.436073 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:53.436030 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:54.098516 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:54.098481 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk"] Nov 01 00:56:54.098773 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:54.098731 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" containerID="cri-o://169ed7395600bafd45344b2ef447d7d943c41b830f1e614fc0b7707fa39d4b01" gracePeriod=30 Nov 01 00:56:54.150539 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:54.150505 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk"] Nov 01 00:56:54.150780 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:54.150743 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" containerID="cri-o://b2c7cdd6d72c6feba80759c81d29622e28d7cf81180aa399f5c3c8d037a35317" gracePeriod=30 Nov 01 00:56:54.154375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:54.154342 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq"] Nov 01 00:56:54.154603 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:54.154574 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" containerID="cri-o://1f5b4724aaa024d165e71ce21c70dbf46561261c0e6cac48a780f391d959fd2e" gracePeriod=30 Nov 01 00:56:54.200068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:54.200032 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt"] Nov 01 00:56:54.200242 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:54.200223 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" containerID="cri-o://c59d3e2f365124168809caa676f7a0c4614a71978fbb5fe392b778e4776c80df" gracePeriod=30 Nov 01 00:56:55.371134 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:55.371091 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk"] Nov 01 00:56:55.371668 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:55.371365 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" containerID="cri-o://2d81f6741fca5cdde8fd998bb97073d11623e2fe13d5588f2fe26c76cc250d54" gracePeriod=30 Nov 01 00:56:55.424539 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:55.424488 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw"] Nov 01 00:56:55.424798 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:55.424767 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" containerID="cri-o://5c2fa5e65c27e3eea5d52269e23da67c6597c4d4f9801349b8334f8f97a84d9d" gracePeriod=30 Nov 01 00:56:55.428186 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:55.428159 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br"] Nov 01 00:56:55.428666 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:55.428638 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" containerID="cri-o://2fa20f32188956356216cda72bdc8c5996769bc6c86fa91954876551803c44b8" gracePeriod=30 Nov 01 00:56:57.351050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:57.351008 6036 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:56:57.351485 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:57.351074 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:56:58.221901 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:58.221868 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:58.930965 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:58.930931 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:59.137961 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.137929 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:59.648760 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.648700 6036 generic.go:356] "Generic (PLEG): container finished" podID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerID="169ed7395600bafd45344b2ef447d7d943c41b830f1e614fc0b7707fa39d4b01" exitCode=0 Nov 01 00:56:59.649578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.649099 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" event={"ID":"00ca2ced-3dc1-479d-ace9-4438014a6952","Type":"ContainerDied","Data":"169ed7395600bafd45344b2ef447d7d943c41b830f1e614fc0b7707fa39d4b01"} Nov 01 00:56:59.652479 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.652436 6036 generic.go:356] "Generic (PLEG): container finished" podID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerID="c59d3e2f365124168809caa676f7a0c4614a71978fbb5fe392b778e4776c80df" exitCode=0 Nov 01 00:56:59.652809 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.652651 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" event={"ID":"dad1844c-94ac-45cf-b5db-13fc278f2f4a","Type":"ContainerDied","Data":"c59d3e2f365124168809caa676f7a0c4614a71978fbb5fe392b778e4776c80df"} Nov 01 00:56:59.656268 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.656215 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" event={"ID":"3e5da06e-9f90-4586-b65e-fa6173da4969","Type":"ContainerDied","Data":"b2c7cdd6d72c6feba80759c81d29622e28d7cf81180aa399f5c3c8d037a35317"} Nov 01 00:56:59.656513 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.656036 6036 generic.go:356] "Generic (PLEG): container finished" podID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerID="b2c7cdd6d72c6feba80759c81d29622e28d7cf81180aa399f5c3c8d037a35317" exitCode=0 Nov 01 00:56:59.659394 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.659216 6036 generic.go:356] "Generic (PLEG): container finished" podID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerID="1f5b4724aaa024d165e71ce21c70dbf46561261c0e6cac48a780f391d959fd2e" exitCode=0 Nov 01 00:56:59.659394 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.659309 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" event={"ID":"2a9454fc-6c85-497d-82c8-2adbbf3c56b7","Type":"ContainerDied","Data":"1f5b4724aaa024d165e71ce21c70dbf46561261c0e6cac48a780f391d959fd2e"} Nov 01 00:56:59.659710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.659693 6036 scope.go:122] "RemoveContainer" containerID="f3c250fde8915e6f0cee6e25a76ccab2a463ea18acbf5a23915e9a4e873772fd" Nov 01 00:56:59.768521 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.753269 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:56:59.792859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.791742 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:56:59.794411 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.794216 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:56:59.799009 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.796838 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:56:59.802505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.802455 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:56:59.864732 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864636 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-certs\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.864732 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864672 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istiod-ca-cert\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.864732 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864695 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-data\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864755 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-certs\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864787 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-token\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-token\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864813 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-socket\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864852 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-envoy\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864883 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-podinfo\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864908 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-certs\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864977 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-envoy\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.864999 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-certs" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.865037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865017 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-token\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-token\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865049 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-token\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-token\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865077 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istiod-ca-cert" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865088 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-data\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865133 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ca2ced-3dc1-479d-ace9-4438014a6952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ca2ced-3dc1-479d-ace9-4438014a6952-istiod-ca-cert\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865186 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5da06e-9f90-4586-b65e-fa6173da4969-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5da06e-9f90-4586-b65e-fa6173da4969-istiod-ca-cert\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865248 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-kube-api-access-cvt74\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-kube-api-access-cvt74\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865253 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-certs" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865318 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-envoy\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865387 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-kube-api-access-f86m7\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-kube-api-access-f86m7\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865426 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-envoy\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865447 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-data" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.865558 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865457 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-credential-socket\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865589 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-credential-socket\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865651 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istiod-ca-cert\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865690 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-token\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-token\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865734 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-podinfo\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865769 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-socket\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865798 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-socket\") pod \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\" (UID: \"dad1844c-94ac-45cf-b5db-13fc278f2f4a\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865862 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-podinfo\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865947 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-certs\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.865985 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-credential-socket\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.866049 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-data\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.866110 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-certs" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.866118 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-kube-api-access-bht4z\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-kube-api-access-bht4z\") pod \"3e5da06e-9f90-4586-b65e-fa6173da4969\" (UID: \"3e5da06e-9f90-4586-b65e-fa6173da4969\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.866151 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-socket\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.866194 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-kube-api-access-2wgx6\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-kube-api-access-2wgx6\") pod \"00ca2ced-3dc1-479d-ace9-4438014a6952\" (UID: \"00ca2ced-3dc1-479d-ace9-4438014a6952\") " Nov 01 00:56:59.866217 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.866222 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-podinfo\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.867169 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.866253 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-data\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.867169 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.866311 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-credential-socket\") pod \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\" (UID: \"2a9454fc-6c85-497d-82c8-2adbbf3c56b7\") " Nov 01 00:56:59.867653 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.867622 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-socket" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.867768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.867650 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.867768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.867674 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.867768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.867689 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.867768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.867704 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.867768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.867718 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.868045 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.867887 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-credential-socket" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.868045 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.867990 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5da06e-9f90-4586-b65e-fa6173da4969-istiod-ca-cert" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:59.870710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.869279 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-credential-socket" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.870710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.869364 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-socket" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.870710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.869436 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-data" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.870710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.869921 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-certs" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.870710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.870248 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-data" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.870710 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.870455 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ca2ced-3dc1-479d-ace9-4438014a6952-istiod-ca-cert" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:59.871360 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.871010 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istiod-ca-cert" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:59.872416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.871670 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-credential-socket" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.872416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.871924 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-data" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.872416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.872090 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-credential-socket" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.872416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.872318 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-socket" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.872416 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.872387 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-podinfo" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:56:59.873847 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.872955 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-socket" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.873847 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.873779 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-kube-api-access-f86m7" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "kube-api-access-f86m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:59.874433 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.874400 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-podinfo" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:56:59.875465 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.875346 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-envoy" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.875465 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.875432 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-token" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:59.876168 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.876134 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-kube-api-access-bht4z" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "kube-api-access-bht4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:59.876580 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.876547 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-envoy" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.876908 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.876878 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-token" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:59.877137 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.877110 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-kube-api-access-2wgx6" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "kube-api-access-2wgx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:59.877236 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.877218 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-envoy" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.877407 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.877386 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-kube-api-access-cvt74" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "kube-api-access-cvt74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:59.877510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.877387 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-podinfo" pod "dad1844c-94ac-45cf-b5db-13fc278f2f4a" (UID: "dad1844c-94ac-45cf-b5db-13fc278f2f4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:56:59.877510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.877465 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-envoy" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:59.877510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.877466 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-token" pod "3e5da06e-9f90-4586-b65e-fa6173da4969" (UID: "3e5da06e-9f90-4586-b65e-fa6173da4969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:59.877510 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.877480 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-token" pod "00ca2ced-3dc1-479d-ace9-4438014a6952" (UID: "00ca2ced-3dc1-479d-ace9-4438014a6952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:59.879483 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.879445 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-podinfo" pod "2a9454fc-6c85-497d-82c8-2adbbf3c56b7" (UID: "2a9454fc-6c85-497d-82c8-2adbbf3c56b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978799 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978833 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ca2ced-3dc1-479d-ace9-4438014a6952-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978849 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5da06e-9f90-4586-b65e-fa6173da4969-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978867 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvt74\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-kube-api-access-cvt74\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978891 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978908 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f86m7\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-kube-api-access-f86m7\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978924 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978938 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978957 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978971 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.978988 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979001 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979021 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979045 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979059 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979072 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979091 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979106 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979121 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bht4z\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-kube-api-access-bht4z\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979134 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ca2ced-3dc1-479d-ace9-4438014a6952-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979153 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wgx6\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-kube-api-access-2wgx6\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979166 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979180 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979193 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979211 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979225 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979239 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979253 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ca2ced-3dc1-479d-ace9-4438014a6952-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979271 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9454fc-6c85-497d-82c8-2adbbf3c56b7-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979307 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5da06e-9f90-4586-b65e-fa6173da4969-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:59.979998 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:56:59.979322 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad1844c-94ac-45cf-b5db-13fc278f2f4a-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:00.321111 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:00.321066 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:57:00.321305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:00.321140 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:57:00.557154 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:00.557121 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 00:57:00.916314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:00.915552 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" Nov 01 00:57:00.935328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:00.920389 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt" event={"ID":"dad1844c-94ac-45cf-b5db-13fc278f2f4a","Type":"ContainerDied","Data":"4369cafa3da3fa29dcff59da67b45002dfe1369bf27504100aee5017cc1a2532"} Nov 01 00:57:00.935328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:00.920487 6036 scope.go:122] "RemoveContainer" containerID="c59d3e2f365124168809caa676f7a0c4614a71978fbb5fe392b778e4776c80df" Nov 01 00:57:01.365326 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.347852 6036 generic.go:356] "Generic (PLEG): container finished" podID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerID="5c2fa5e65c27e3eea5d52269e23da67c6597c4d4f9801349b8334f8f97a84d9d" exitCode=0 Nov 01 00:57:01.639753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.639462 6036 generic.go:356] "Generic (PLEG): container finished" podID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerID="2fa20f32188956356216cda72bdc8c5996769bc6c86fa91954876551803c44b8" exitCode=0 Nov 01 00:57:01.645808 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.643035 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" event={"ID":"4855a861-3590-4e06-85d7-0bf38f8b463b","Type":"ContainerDied","Data":"5c2fa5e65c27e3eea5d52269e23da67c6597c4d4f9801349b8334f8f97a84d9d"} Nov 01 00:57:01.645808 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.643104 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt"] Nov 01 00:57:01.645808 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.643135 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-8xljt"] Nov 01 00:57:01.645808 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.643162 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" event={"ID":"21aa9c83-33c1-4471-9bfc-8d0eab7459ae","Type":"ContainerDied","Data":"2fa20f32188956356216cda72bdc8c5996769bc6c86fa91954876551803c44b8"} Nov 01 00:57:01.656308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.655815 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" Nov 01 00:57:01.672235 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.672193 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk" event={"ID":"3e5da06e-9f90-4586-b65e-fa6173da4969","Type":"ContainerDied","Data":"a05974e50e51153b6be03fae3513751bcc05db36c46b12f5db1ee42af96c769f"} Nov 01 00:57:01.672543 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.672518 6036 scope.go:122] "RemoveContainer" containerID="b2c7cdd6d72c6feba80759c81d29622e28d7cf81180aa399f5c3c8d037a35317" Nov 01 00:57:01.847329 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.847269 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" event={"ID":"2a9454fc-6c85-497d-82c8-2adbbf3c56b7","Type":"ContainerDied","Data":"cea208bc49c3d21cbbbbbc47f5546fb8d6dcb6535f7400c9ef9bdd5f51158593"} Nov 01 00:57:01.848702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.848070 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq" Nov 01 00:57:01.879205 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.879175 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" Nov 01 00:57:01.880080 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.879447 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk" event={"ID":"00ca2ced-3dc1-479d-ace9-4438014a6952","Type":"ContainerDied","Data":"f45d677e8e6809b6b31109fd967fff4d2ea7a7b7bd245dd8802be78fe9116a04"} Nov 01 00:57:01.892690 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.892617 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:57:01.950887 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:01.950855 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk"] Nov 01 00:57:02.043761 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.043727 6036 generic.go:356] "Generic (PLEG): container finished" podID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerID="2d81f6741fca5cdde8fd998bb97073d11623e2fe13d5588f2fe26c76cc250d54" exitCode=0 Nov 01 00:57:02.044124 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.044097 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" event={"ID":"e7626ba6-6131-4a6a-ba8e-08531f8a1daf","Type":"ContainerDied","Data":"2d81f6741fca5cdde8fd998bb97073d11623e2fe13d5588f2fe26c76cc250d54"} Nov 01 00:57:02.060553 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.060523 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f9bkk"] Nov 01 00:57:02.093043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.093015 6036 scope.go:122] "RemoveContainer" containerID="1f5b4724aaa024d165e71ce21c70dbf46561261c0e6cac48a780f391d959fd2e" Nov 01 00:57:02.111100 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.111062 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-envoy\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.116317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.111421 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-token\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-token\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.116689 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.116661 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-credential-socket\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.116893 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.116875 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-data\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.117062 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.117043 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-certs\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.127413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.127380 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-kube-api-access-rxpqj\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-kube-api-access-rxpqj\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.127717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.127697 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-podinfo\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.127903 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.127885 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istiod-ca-cert\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.128037 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.128019 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-socket\") pod \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\" (UID: \"21aa9c83-33c1-4471-9bfc-8d0eab7459ae\") " Nov 01 00:57:02.137413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.125141 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-credential-socket" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.137413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.125347 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-data" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.137413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.130689 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istiod-ca-cert" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:02.137413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.130982 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-certs" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.158470 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.158394 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq"] Nov 01 00:57:02.178814 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.178778 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-socket" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.206179 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.206143 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-envoy" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.228090 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.228054 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-podinfo" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:02.237840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.231876 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.237840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.231914 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.237840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.231931 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.237840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.231945 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.237840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.231959 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.237840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.231974 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.237840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.231988 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.237840 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.236763 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:57:02.238611 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.238542 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-fzcsq"] Nov 01 00:57:02.238923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.238895 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-token" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:02.270620 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.270587 6036 scope.go:122] "RemoveContainer" containerID="169ed7395600bafd45344b2ef447d7d943c41b830f1e614fc0b7707fa39d4b01" Nov 01 00:57:02.273506 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.273466 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-kube-api-access-rxpqj" pod "21aa9c83-33c1-4471-9bfc-8d0eab7459ae" (UID: "21aa9c83-33c1-4471-9bfc-8d0eab7459ae"). InnerVolumeSpecName "kube-api-access-rxpqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:02.307405 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.307378 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:57:02.333331 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.333263 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-token\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-token\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.333524 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.333479 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-socket\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.333601 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.333555 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-credential-socket\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.333668 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.333607 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-kube-api-access-mln67\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-kube-api-access-mln67\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.333733 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.333698 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4855a861-3590-4e06-85d7-0bf38f8b463b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855a861-3590-4e06-85d7-0bf38f8b463b-istiod-ca-cert\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.333816 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.333798 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-data\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.334024 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.334008 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-certs\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.334227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.333876 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-credential-socket" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.334227 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.334162 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4855a861-3590-4e06-85d7-0bf38f8b463b-istiod-ca-cert" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:02.334432 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.334358 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-podinfo\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.335050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.334532 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-envoy\") pod \"4855a861-3590-4e06-85d7-0bf38f8b463b\" (UID: \"4855a861-3590-4e06-85d7-0bf38f8b463b\") " Nov 01 00:57:02.335050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.334764 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-certs" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.335449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.335335 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxpqj\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-kube-api-access-rxpqj\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.335449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.335342 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-socket" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.335449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.335361 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.335449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.335380 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855a861-3590-4e06-85d7-0bf38f8b463b-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.335449 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.335395 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21aa9c83-33c1-4471-9bfc-8d0eab7459ae-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.335749 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.335409 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.335885 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.335851 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-data" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.336486 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.336458 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-token" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:02.337397 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.337374 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-kube-api-access-mln67" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "kube-api-access-mln67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:02.337858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.337842 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-podinfo" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:02.338356 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.338340 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-envoy" pod "4855a861-3590-4e06-85d7-0bf38f8b463b" (UID: "4855a861-3590-4e06-85d7-0bf38f8b463b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.338445 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.338386 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:57:02.338445 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.338423 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:57:02.375023 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.374942 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk"] Nov 01 00:57:02.381482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.381450 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-n5vkk"] Nov 01 00:57:02.436236 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436110 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-data\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436236 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436225 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-kube-api-access-4fffr\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-kube-api-access-4fffr\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436542 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436326 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-socket\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436542 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436366 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-podinfo\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436542 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436395 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-envoy\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436542 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436432 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istiod-ca-cert\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436542 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436479 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-token\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-token\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436542 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436478 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-data" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.436845 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436554 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-certs\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436845 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436651 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-credential-socket\") pod \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\" (UID: \"e7626ba6-6131-4a6a-ba8e-08531f8a1daf\") " Nov 01 00:57:02.436845 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.436727 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-socket" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.437380 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437324 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-certs" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.437694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437556 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-credential-socket" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.437694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437622 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.437863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437770 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.437863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437792 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.437863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437806 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.437863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437820 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.437863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437834 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.437863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437848 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855a861-3590-4e06-85d7-0bf38f8b463b-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.437863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.437863 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mln67\" (UniqueName: \"kubernetes.io/projected/4855a861-3590-4e06-85d7-0bf38f8b463b-kube-api-access-mln67\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.438193 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.438064 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istiod-ca-cert" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:02.439239 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.439208 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-podinfo" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:02.439351 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.439298 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-envoy" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:02.439830 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.439806 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-token" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:02.440060 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.440040 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-kube-api-access-4fffr" pod "e7626ba6-6131-4a6a-ba8e-08531f8a1daf" (UID: "e7626ba6-6131-4a6a-ba8e-08531f8a1daf"). InnerVolumeSpecName "kube-api-access-4fffr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:02.539937 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.539901 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fffr\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-kube-api-access-4fffr\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.539937 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.539935 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.539937 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.539948 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.540216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.539963 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.540216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.539978 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.540216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.539991 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:02.540216 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:02.540002 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7626ba6-6131-4a6a-ba8e-08531f8a1daf-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:03.055574 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.055543 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" Nov 01 00:57:03.055574 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.055553 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk" event={"ID":"e7626ba6-6131-4a6a-ba8e-08531f8a1daf","Type":"ContainerDied","Data":"ed9927e019f5c5b633b87cc331a1ab26023dba5e3d8d73a9ea393e918837d3a7"} Nov 01 00:57:03.055842 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.055649 6036 scope.go:122] "RemoveContainer" containerID="2d81f6741fca5cdde8fd998bb97073d11623e2fe13d5588f2fe26c76cc250d54" Nov 01 00:57:03.058361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.058333 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" event={"ID":"4855a861-3590-4e06-85d7-0bf38f8b463b","Type":"ContainerDied","Data":"b77391aa96fc28be03c2a5b6815a042a35d04fe396f01514804de9dd9356d6af"} Nov 01 00:57:03.058551 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.058345 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw" Nov 01 00:57:03.059909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.059887 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" event={"ID":"21aa9c83-33c1-4471-9bfc-8d0eab7459ae","Type":"ContainerDied","Data":"2c704aa09aac6f73a488cbc7400197b9141600af14823f1d6a539b3859a3fe58"} Nov 01 00:57:03.059994 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.059948 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br" Nov 01 00:57:03.068627 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.068513 6036 scope.go:122] "RemoveContainer" containerID="5c2fa5e65c27e3eea5d52269e23da67c6597c4d4f9801349b8334f8f97a84d9d" Nov 01 00:57:03.079156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.079136 6036 scope.go:122] "RemoveContainer" containerID="2fa20f32188956356216cda72bdc8c5996769bc6c86fa91954876551803c44b8" Nov 01 00:57:03.234898 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.234852 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw"] Nov 01 00:57:03.239006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.238979 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-2qhpw"] Nov 01 00:57:03.251553 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.251523 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" path="/var/lib/kubelet/pods/00ca2ced-3dc1-479d-ace9-4438014a6952/volumes" Nov 01 00:57:03.252074 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.252053 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" path="/var/lib/kubelet/pods/2a9454fc-6c85-497d-82c8-2adbbf3c56b7/volumes" Nov 01 00:57:03.252619 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.252597 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" path="/var/lib/kubelet/pods/3e5da06e-9f90-4586-b65e-fa6173da4969/volumes" Nov 01 00:57:03.253419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.253400 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" path="/var/lib/kubelet/pods/4855a861-3590-4e06-85d7-0bf38f8b463b/volumes" Nov 01 00:57:03.253879 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.253861 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" path="/var/lib/kubelet/pods/dad1844c-94ac-45cf-b5db-13fc278f2f4a/volumes" Nov 01 00:57:03.352118 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.352033 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk"] Nov 01 00:57:03.355526 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.355503 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-h7zhk"] Nov 01 00:57:03.469116 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.469080 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br"] Nov 01 00:57:03.472852 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:03.472826 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-rl7br"] Nov 01 00:57:04.116067 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116039 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5"] Nov 01 00:57:04.116688 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116674 6036 cpu_manager.go:414] "containerMap: removing container" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" Nov 01 00:57:04.116756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116689 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" Nov 01 00:57:04.116756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116702 6036 cpu_manager.go:414] "containerMap: removing container" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" Nov 01 00:57:04.116756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116707 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" Nov 01 00:57:04.116756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116740 6036 cpu_manager.go:414] "containerMap: removing container" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" Nov 01 00:57:04.116756 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116745 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116762 6036 cpu_manager.go:414] "containerMap: removing container" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116768 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116781 6036 cpu_manager.go:414] "containerMap: removing container" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116785 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116795 6036 cpu_manager.go:414] "containerMap: removing container" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116800 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116808 6036 cpu_manager.go:414] "containerMap: removing container" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116812 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116912 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116924 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116931 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ca2ced-3dc1-479d-ace9-4438014a6952" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116938 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4855a861-3590-4e06-85d7-0bf38f8b463b" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116943 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116948 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbbbc8a-1d74-465c-9e10-40840abceb51" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116957 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad1844c-94ac-45cf-b5db-13fc278f2f4a" containerName="istio-proxy" Nov 01 00:57:04.117006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.116962 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9454fc-6c85-497d-82c8-2adbbf3c56b7" containerName="istio-proxy" Nov 01 00:57:04.122137 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.122115 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.163345 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.163276 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4510bd21-cf38-42c7-9fe2-cf19a021af4d-openshift-service-ca\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.163526 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.163507 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.163682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.163667 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-metrics-certs\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.163720 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.163704 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4srh\" (UniqueName: \"kubernetes.io/projected/4510bd21-cf38-42c7-9fe2-cf19a021af4d-kube-api-access-g4srh\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.179090 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.179059 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b112c158-serving-cert\"" Nov 01 00:57:04.183082 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.183050 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5"] Nov 01 00:57:04.264797 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.264755 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-metrics-certs\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.265010 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.264804 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4srh\" (UniqueName: \"kubernetes.io/projected/4510bd21-cf38-42c7-9fe2-cf19a021af4d-kube-api-access-g4srh\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.265010 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.264848 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4510bd21-cf38-42c7-9fe2-cf19a021af4d-openshift-service-ca\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.265010 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.264975 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.265209 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:57:04.265166 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-serving-cert: secret "maas-api-e2e-mt-b112c158-serving-cert" not found Nov 01 00:57:04.265304 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:57:04.265272 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls podName:4510bd21-cf38-42c7-9fe2-cf19a021af4d nodeName:}" failed. No retries permitted until 2026-11-01 00:57:04.765247027 +0000 UTC m=+1969.640468029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls") pod "maas-api-e2e-mt-b112c158-799d88785c-vsdc5" (UID: "4510bd21-cf38-42c7-9fe2-cf19a021af4d") : secret "maas-api-e2e-mt-b112c158-serving-cert" not found Nov 01 00:57:04.265790 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.265765 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4510bd21-cf38-42c7-9fe2-cf19a021af4d-openshift-service-ca\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.267644 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.267617 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-metrics-certs\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.327741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.327710 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4srh\" (UniqueName: \"kubernetes.io/projected/4510bd21-cf38-42c7-9fe2-cf19a021af4d-kube-api-access-g4srh\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.442546 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.442471 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:04.774183 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.774087 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:04.777332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.777249 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:04.777507 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:04.777377 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls\") pod \"maas-api-e2e-mt-b112c158-799d88785c-vsdc5\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:05.036894 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:05.036838 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:05.253419 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:05.253389 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21aa9c83-33c1-4471-9bfc-8d0eab7459ae" path="/var/lib/kubelet/pods/21aa9c83-33c1-4471-9bfc-8d0eab7459ae/volumes" Nov 01 00:57:05.253986 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:05.253958 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7626ba6-6131-4a6a-ba8e-08531f8a1daf" path="/var/lib/kubelet/pods/e7626ba6-6131-4a6a-ba8e-08531f8a1daf/volumes" Nov 01 00:57:05.423091 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:05.423063 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5"] Nov 01 00:57:05.424468 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:57:05.424433 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4510bd21_cf38_42c7_9fe2_cf19a021af4d.slice/crio-272d94ee461072f34ac4ca1370e2e041f474cc52cd126893e262d88f549d2988 WatchSource:0}: Error finding container 272d94ee461072f34ac4ca1370e2e041f474cc52cd126893e262d88f549d2988: Status 404 returned error can't find the container with id 272d94ee461072f34ac4ca1370e2e041f474cc52cd126893e262d88f549d2988 Nov 01 00:57:06.086243 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:06.086181 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" event={"ID":"4510bd21-cf38-42c7-9fe2-cf19a021af4d","Type":"ContainerStarted","Data":"272d94ee461072f34ac4ca1370e2e041f474cc52cd126893e262d88f549d2988"} Nov 01 00:57:06.480860 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:06.480826 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:06.695366 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:06.695220 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:07.095725 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:07.095689 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" event={"ID":"4510bd21-cf38-42c7-9fe2-cf19a021af4d","Type":"ContainerStarted","Data":"69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a"} Nov 01 00:57:07.096128 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:07.095799 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:09.684897 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:09.684862 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:10.141546 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:10.141497 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" podStartSLOduration=5.2671952619999995 podStartE2EDuration="6.141481424s" totalImagesPullingTime="874.286162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:07.218523056 +0000 UTC m=+1972.093744072" watchObservedRunningTime="2026-11-01 00:57:10.141481424 +0000 UTC m=+1975.016702476" Nov 01 00:57:10.144530 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:10.144503 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9"] Nov 01 00:57:10.144745 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:10.144724 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" containerID="cri-o://b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15" gracePeriod=30 Nov 01 00:57:13.107694 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:13.107660 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 00:57:14.827226 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:14.827189 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:15.427753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.427680 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:57:15.542074 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542041 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/094e131e-acdd-4ef3-ae8f-a499884df745-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094e131e-acdd-4ef3-ae8f-a499884df745-istio-podinfo\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.542269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542082 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-envoy\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.542269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542218 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-data\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.542450 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542376 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-kube-api-access-bmtz8\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-kube-api-access-bmtz8\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.542505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542470 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-certs\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.543043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542579 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-socket\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.543043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542612 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-data" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.543043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542625 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094e131e-acdd-4ef3-ae8f-a499884df745-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094e131e-acdd-4ef3-ae8f-a499884df745-istiod-ca-cert\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.543043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542684 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-credential-socket\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.543043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542853 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-socket" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.543043 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.542856 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-istio-token\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-istio-token\") pod \"094e131e-acdd-4ef3-ae8f-a499884df745\" (UID: \"094e131e-acdd-4ef3-ae8f-a499884df745\") " Nov 01 00:57:15.543638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.543067 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094e131e-acdd-4ef3-ae8f-a499884df745-istiod-ca-cert" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:15.543638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.543162 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-certs" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.543638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.543183 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-credential-socket" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.543813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.543715 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.543813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.543731 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.543813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.543740 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094e131e-acdd-4ef3-ae8f-a499884df745-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.543813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.543751 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.543813 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.543759 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.544852 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.544819 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/094e131e-acdd-4ef3-ae8f-a499884df745-istio-podinfo" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:15.545375 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.545337 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-kube-api-access-bmtz8" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "kube-api-access-bmtz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:15.545496 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.545472 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-istio-token" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:15.545819 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.545799 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-envoy" pod "094e131e-acdd-4ef3-ae8f-a499884df745" (UID: "094e131e-acdd-4ef3-ae8f-a499884df745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.646083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.646036 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmtz8\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-kube-api-access-bmtz8\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.646083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.646084 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094e131e-acdd-4ef3-ae8f-a499884df745-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.646314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.646099 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094e131e-acdd-4ef3-ae8f-a499884df745-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.646314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:15.646115 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094e131e-acdd-4ef3-ae8f-a499884df745-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:16.159184 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.159144 6036 generic.go:356] "Generic (PLEG): container finished" podID="094e131e-acdd-4ef3-ae8f-a499884df745" containerID="b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15" exitCode=0 Nov 01 00:57:16.159704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.159202 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" Nov 01 00:57:16.159704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.159222 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" event={"ID":"094e131e-acdd-4ef3-ae8f-a499884df745","Type":"ContainerDied","Data":"b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15"} Nov 01 00:57:16.159704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.159313 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9" event={"ID":"094e131e-acdd-4ef3-ae8f-a499884df745","Type":"ContainerDied","Data":"f36306f0ee2e7937b5241413d29afb86c45ea5e234e10fd1df35a8bb72e0419b"} Nov 01 00:57:16.159704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.159351 6036 scope.go:122] "RemoveContainer" containerID="b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15" Nov 01 00:57:16.171540 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.171520 6036 scope.go:122] "RemoveContainer" containerID="b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15" Nov 01 00:57:16.171886 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:57:16.171865 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15\": container with ID starting with b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15 not found: ID does not exist" containerID="b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15" Nov 01 00:57:16.171967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.171900 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15"} err="failed to get container status \"b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15\": rpc error: code = NotFound desc = could not find container \"b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15\": container with ID starting with b078a0ed0e3d68cdc84d85a1cf958498d421feae87cadc95e57b5ed654f52d15 not found: ID does not exist" Nov 01 00:57:16.382562 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.382520 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9"] Nov 01 00:57:16.433347 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:16.433313 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-dfhd9"] Nov 01 00:57:17.251817 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:17.251773 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" path="/var/lib/kubelet/pods/094e131e-acdd-4ef3-ae8f-a499884df745/volumes" Nov 01 00:57:19.189051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:19.189012 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:19.413932 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:19.413894 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:19.507996 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:57:19.507904 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:57:19.684631 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:19.684593 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:24.677731 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:24.677610 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:29.555954 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:29.555917 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:31.724308 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:31.724261 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:34.664168 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:34.664136 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:35.541213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:35.541174 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:36.225992 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.225953 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:36.451048 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.450991 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf"] Nov 01 00:57:36.451959 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.451935 6036 cpu_manager.go:414] "containerMap: removing container" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" Nov 01 00:57:36.452145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.452131 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" Nov 01 00:57:36.475418 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.475386 6036 cpu_manager.go:414] "containerMap: removing container" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" Nov 01 00:57:36.475697 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.475679 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" Nov 01 00:57:36.476145 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.476090 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e131e-acdd-4ef3-ae8f-a499884df745" containerName="istio-proxy" Nov 01 00:57:36.476266 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.476255 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5da06e-9f90-4586-b65e-fa6173da4969" containerName="istio-proxy" Nov 01 00:57:36.481036 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.481007 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.518963 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.518885 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf"] Nov 01 00:57:36.572279 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.570947 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-44c9bf-openshift-default-dockercfg-hm5pk\"" Nov 01 00:57:36.632362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.606655 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-socket\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.632726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.632683 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-certs\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.632967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.632948 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-data\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.633149 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.633130 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsr9l\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-kube-api-access-nsr9l\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.633268 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.632384 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:36.633502 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.633483 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-envoy\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.633738 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.633721 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-istio-token\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.633924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.633908 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8844a5a-492a-4534-a3af-07def8126146-istiod-ca-cert\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.634112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.634095 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8844a5a-492a-4534-a3af-07def8126146-istio-podinfo\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.634331 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.634315 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-credential-socket\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.736875 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-credential-socket\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737156 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.737140 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-socket\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737429 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.737189 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-certs\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737429 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.737241 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-data\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737429 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.737276 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsr9l\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-kube-api-access-nsr9l\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737429 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.737359 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-envoy\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737632 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.737447 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-istio-token\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737632 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.737487 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8844a5a-492a-4534-a3af-07def8126146-istiod-ca-cert\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.737632 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.737527 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8844a5a-492a-4534-a3af-07def8126146-istio-podinfo\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.738498 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.738466 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-socket\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.738664 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.738514 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-data\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.739138 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.739114 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-credential-socket\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.739277 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.739253 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-certs\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.739420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.739398 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8844a5a-492a-4534-a3af-07def8126146-istiod-ca-cert\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.741622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.741597 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8844a5a-492a-4534-a3af-07def8126146-istio-podinfo\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.742243 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.742217 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-envoy\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.810670 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.810626 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsr9l\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-kube-api-access-nsr9l\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.811586 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.811548 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-istio-token\") pod \"e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:36.861704 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:36.861663 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:37.018965 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:37.018877 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:37.234215 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:37.234177 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:37.305242 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:37.305206 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf"] Nov 01 00:57:37.305586 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:57:37.305567 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8844a5a_492a_4534_a3af_07def8126146.slice/crio-f4795dec9ded84c72cfbab19e7dd9fd79ef3b1fc23d45bc49d24626e0cb7d016 WatchSource:0}: Error finding container f4795dec9ded84c72cfbab19e7dd9fd79ef3b1fc23d45bc49d24626e0cb7d016: Status 404 returned error can't find the container with id f4795dec9ded84c72cfbab19e7dd9fd79ef3b1fc23d45bc49d24626e0cb7d016 Nov 01 00:57:37.308271 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:37.308231 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:37.308404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:37.308337 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:37.308404 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:37.308381 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:37.366427 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:37.366396 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" event={"ID":"d8844a5a-492a-4534-a3af-07def8126146","Type":"ContainerStarted","Data":"f4795dec9ded84c72cfbab19e7dd9fd79ef3b1fc23d45bc49d24626e0cb7d016"} Nov 01 00:57:38.179082 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.179037 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9"] Nov 01 00:57:38.183549 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.183520 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.237622 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.237583 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ba5b10-openshift-default-dockercfg-rcckb\"" Nov 01 00:57:38.284935 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.284895 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-certs\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.285178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.284946 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f402208-c342-4429-8b0b-3259e410455d-istio-podinfo\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.285178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.285100 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-socket\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.285178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.285130 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-envoy\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.285397 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.285233 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-credential-socket\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.285546 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.285525 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f402208-c342-4429-8b0b-3259e410455d-istiod-ca-cert\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.285640 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.285561 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-istio-token\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.285640 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.285586 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklww\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-kube-api-access-fklww\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.285640 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.285611 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-data\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.289735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.289707 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9"] Nov 01 00:57:38.363053 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.363014 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:38.379213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.379170 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" event={"ID":"d8844a5a-492a-4534-a3af-07def8126146","Type":"ContainerStarted","Data":"d57fbafa75c687c35b4d1cf3e902bcc2ddc8189c75569512625a567fdcd317d8"} Nov 01 00:57:38.387882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.387848 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-certs\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.387882 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.387901 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f402208-c342-4429-8b0b-3259e410455d-istio-podinfo\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.388195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.387993 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-socket\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.388195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.388017 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-envoy\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.388195 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.388044 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-credential-socket\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.388495 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.388273 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-certs\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.389928 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.388408 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-socket\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.390532 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.390508 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-credential-socket\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.391051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.391010 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f402208-c342-4429-8b0b-3259e410455d-istiod-ca-cert\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.392477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.391065 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-istio-token\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.392477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.391097 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklww\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-kube-api-access-fklww\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.392477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.391124 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-data\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.392477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.391431 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-data\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.392477 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.392114 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f402208-c342-4429-8b0b-3259e410455d-istiod-ca-cert\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.394386 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.394361 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f402208-c342-4429-8b0b-3259e410455d-istio-podinfo\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.394875 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.394852 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-envoy\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.455243 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.455149 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-istio-token\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.455482 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.455458 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklww\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-kube-api-access-fklww\") pod \"e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.498696 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.498666 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:38.506040 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.505996 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podStartSLOduration=2.5059840639999997 podStartE2EDuration="2.505984064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:38.505751563 +0000 UTC m=+2003.380972577" watchObservedRunningTime="2026-11-01 00:57:38.505984064 +0000 UTC m=+2003.381205112" Nov 01 00:57:38.790361 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.790325 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:38.862880 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.862843 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:38.869181 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.869151 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:38.940649 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:57:38.940616 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f402208_c342_4429_8b0b_3259e410455d.slice/crio-f292509764265523cdf193c87f2c90a0060994f46db70ed95974f52a48f05e33 WatchSource:0}: Error finding container f292509764265523cdf193c87f2c90a0060994f46db70ed95974f52a48f05e33: Status 404 returned error can't find the container with id f292509764265523cdf193c87f2c90a0060994f46db70ed95974f52a48f05e33 Nov 01 00:57:38.941616 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.941585 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9"] Nov 01 00:57:38.943721 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.943684 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:38.943818 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.943754 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:38.943818 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:38.943784 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:39.388274 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.388230 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" event={"ID":"4f402208-c342-4429-8b0b-3259e410455d","Type":"ContainerStarted","Data":"b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea"} Nov 01 00:57:39.388785 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.388363 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" event={"ID":"4f402208-c342-4429-8b0b-3259e410455d","Type":"ContainerStarted","Data":"f292509764265523cdf193c87f2c90a0060994f46db70ed95974f52a48f05e33"} Nov 01 00:57:39.388909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.388862 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:39.389923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.389898 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:57:39.499109 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.499078 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:39.514606 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.514553 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" podStartSLOduration=1.5145388789999998 podStartE2EDuration="1.514538879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:39.511707155 +0000 UTC m=+2004.386928170" watchObservedRunningTime="2026-11-01 00:57:39.514538879 +0000 UTC m=+2004.389759895" Nov 01 00:57:39.863413 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.863377 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6"] Nov 01 00:57:39.867833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.867805 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:39.913261 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.913223 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:39.913261 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.913258 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:39.913508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.913414 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2t4\" (UniqueName: \"kubernetes.io/projected/db4381d5-90f1-407d-9130-7ce9910ce004-kube-api-access-dl2t4\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:39.913552 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.913526 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4381d5-90f1-407d-9130-7ce9910ce004-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:39.922174 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.922137 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-27dee6-serving-cert\"" Nov 01 00:57:39.923498 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:39.923474 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6"] Nov 01 00:57:40.014948 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.014917 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.014948 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.014952 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.015213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.014998 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2t4\" (UniqueName: \"kubernetes.io/projected/db4381d5-90f1-407d-9130-7ce9910ce004-kube-api-access-dl2t4\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.015213 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.015083 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4381d5-90f1-407d-9130-7ce9910ce004-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.015213 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:57:40.015087 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-serving-cert: secret "maas-api-e2e-shared-a-w1-27dee6-serving-cert" not found Nov 01 00:57:40.015213 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:57:40.015159 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls podName:db4381d5-90f1-407d-9130-7ce9910ce004 nodeName:}" failed. No retries permitted until 2026-11-01 00:57:40.515137055 +0000 UTC m=+2005.390358052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls") pod "maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" (UID: "db4381d5-90f1-407d-9130-7ce9910ce004") : secret "maas-api-e2e-shared-a-w1-27dee6-serving-cert" not found Nov 01 00:57:40.015921 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.015831 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4381d5-90f1-407d-9130-7ce9910ce004-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.017763 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.017738 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.078068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.078031 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2t4\" (UniqueName: \"kubernetes.io/projected/db4381d5-90f1-407d-9130-7ce9910ce004-kube-api-access-dl2t4\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.132985 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.132883 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:40.391682 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.391581 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch"] Nov 01 00:57:40.397129 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.397073 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" containerID="cri-o://6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792" gracePeriod=30 Nov 01 00:57:40.422854 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.422816 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:40.505508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.505477 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:40.523448 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.523416 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.526112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.526085 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:40.649115 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.649035 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:40.784403 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:40.784368 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:41.059855 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:41.059821 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:41.172512 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:41.172483 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6"] Nov 01 00:57:41.172747 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:57:41.172724 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4381d5_90f1_407d_9130_7ce9910ce004.slice/crio-cbc44b9292f02580481423f71e33ec0e13787bc75e511debea7ed286b70b8bca WatchSource:0}: Error finding container cbc44b9292f02580481423f71e33ec0e13787bc75e511debea7ed286b70b8bca: Status 404 returned error can't find the container with id cbc44b9292f02580481423f71e33ec0e13787bc75e511debea7ed286b70b8bca Nov 01 00:57:41.174357 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:41.174275 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:57:41.422748 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:41.422661 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" event={"ID":"db4381d5-90f1-407d-9130-7ce9910ce004","Type":"ContainerStarted","Data":"cbc44b9292f02580481423f71e33ec0e13787bc75e511debea7ed286b70b8bca"} Nov 01 00:57:41.423179 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:41.423155 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:41.423918 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:41.423897 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:57:42.441859 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:42.441818 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" event={"ID":"db4381d5-90f1-407d-9130-7ce9910ce004","Type":"ContainerStarted","Data":"7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e"} Nov 01 00:57:42.442386 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:42.441991 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:42.563591 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:42.563536 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" podStartSLOduration=2.727193461 podStartE2EDuration="3.563520107s" totalImagesPullingTime="836.326646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:42.561184696 +0000 UTC m=+2007.436405710" watchObservedRunningTime="2026-11-01 00:57:42.563520107 +0000 UTC m=+2007.438741122" Nov 01 00:57:42.580244 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:42.580209 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:42.812113 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:42.812074 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:43.251629 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:43.251533 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:43.479931 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:43.479891 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:43.695578 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:43.695546 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:45.719058 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.718978 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:45.720814 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.720792 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:57:45.809698 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.809662 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-envoy\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.809909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.809812 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-socket\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.809909 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.809877 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7cfb12-70c6-4469-9c8a-5997786d2125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7cfb12-70c6-4469-9c8a-5997786d2125-istiod-ca-cert\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.810016 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.809939 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-data\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.810103 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.810068 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-credential-socket\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.810237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.810199 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-certs\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.810362 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.810345 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-podinfo\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.810436 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.810387 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-kube-api-access-rmzxq\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-kube-api-access-rmzxq\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.810436 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.810427 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-token\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-token\") pod \"9d7cfb12-70c6-4469-9c8a-5997786d2125\" (UID: \"9d7cfb12-70c6-4469-9c8a-5997786d2125\") " Nov 01 00:57:45.810721 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.810696 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-socket" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:45.811505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.811189 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-certs" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:45.811505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.811401 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-data" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:45.811505 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.811442 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-credential-socket" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:45.811702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.811643 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7cfb12-70c6-4469-9c8a-5997786d2125-istiod-ca-cert" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:45.812151 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.812127 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.812241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.812157 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7cfb12-70c6-4469-9c8a-5997786d2125-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.812241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.812173 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.812241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.812188 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.812241 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.812205 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.814159 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.814140 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-envoy" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:45.815637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.815614 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-podinfo" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:45.815759 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.815740 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-kube-api-access-rmzxq" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "kube-api-access-rmzxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:45.817808 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.817782 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-token" pod "9d7cfb12-70c6-4469-9c8a-5997786d2125" (UID: "9d7cfb12-70c6-4469-9c8a-5997786d2125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:45.914269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.914224 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.914269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.914262 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzxq\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-kube-api-access-rmzxq\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.914556 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.914278 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.914556 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.914414 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7cfb12-70c6-4469-9c8a-5997786d2125-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.928846 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:45.928750 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:46.146517 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.146480 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:46.209050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.208967 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh"] Nov 01 00:57:46.209688 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.209672 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" Nov 01 00:57:46.209759 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.209690 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" Nov 01 00:57:46.209811 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.209802 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerName="istio-proxy" Nov 01 00:57:46.213556 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.213541 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:46.267385 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.267332 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-44c9bf\"" Nov 01 00:57:46.267659 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.267631 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-44c9bf-dockercfg-pb5vn\"" Nov 01 00:57:46.271739 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.271711 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh"] Nov 01 00:57:46.340140 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.338095 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zcm\" (UniqueName: \"kubernetes.io/projected/85dd805a-c7b1-49fb-9767-bc0309882f04-kube-api-access-75zcm\") pod \"payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh\" (UID: \"85dd805a-c7b1-49fb-9767-bc0309882f04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:46.346473 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.346429 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dd805a-c7b1-49fb-9767-bc0309882f04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh\" (UID: \"85dd805a-c7b1-49fb-9767-bc0309882f04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:46.406268 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.406232 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp"] Nov 01 00:57:46.480354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.472114 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:46.480354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.474165 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zcm\" (UniqueName: \"kubernetes.io/projected/85dd805a-c7b1-49fb-9767-bc0309882f04-kube-api-access-75zcm\") pod \"payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh\" (UID: \"85dd805a-c7b1-49fb-9767-bc0309882f04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:46.480354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.474309 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dd805a-c7b1-49fb-9767-bc0309882f04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh\" (UID: \"85dd805a-c7b1-49fb-9767-bc0309882f04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:46.480354 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.475669 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dd805a-c7b1-49fb-9767-bc0309882f04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh\" (UID: \"85dd805a-c7b1-49fb-9767-bc0309882f04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:46.486420 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.486311 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp"] Nov 01 00:57:46.515727 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.515693 6036 generic.go:356] "Generic (PLEG): container finished" podID="9d7cfb12-70c6-4469-9c8a-5997786d2125" containerID="6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792" exitCode=0 Nov 01 00:57:46.515898 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.515738 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" event={"ID":"9d7cfb12-70c6-4469-9c8a-5997786d2125","Type":"ContainerDied","Data":"6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792"} Nov 01 00:57:46.515898 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.515787 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" event={"ID":"9d7cfb12-70c6-4469-9c8a-5997786d2125","Type":"ContainerDied","Data":"7b31a7c3bb9d123e76f2a8d3e326626e0121fc349330d9bab4369395466f4e69"} Nov 01 00:57:46.515898 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.515816 6036 scope.go:122] "RemoveContainer" containerID="6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792" Nov 01 00:57:46.516099 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.515985 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch" Nov 01 00:57:46.533706 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.533679 6036 scope.go:122] "RemoveContainer" containerID="6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792" Nov 01 00:57:46.534037 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:57:46.534003 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792\": container with ID starting with 6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792 not found: ID does not exist" containerID="6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792" Nov 01 00:57:46.534187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.534048 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792"} err="failed to get container status \"6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792\": rpc error: code = NotFound desc = could not find container \"6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792\": container with ID starting with 6744fb481ce42f7c2b2f0eb87d7061449e9864c7b22156822c6d842f4edeb792 not found: ID does not exist" Nov 01 00:57:46.538443 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.538417 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zcm\" (UniqueName: \"kubernetes.io/projected/85dd805a-c7b1-49fb-9767-bc0309882f04-kube-api-access-75zcm\") pod \"payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh\" (UID: \"85dd805a-c7b1-49fb-9767-bc0309882f04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:46.575650 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.575609 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664423ef-4053-4db9-b4a4-85a93fb2e4fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp\" (UID: \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:46.575898 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.575823 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsw6l\" (UniqueName: \"kubernetes.io/projected/664423ef-4053-4db9-b4a4-85a93fb2e4fd-kube-api-access-gsw6l\") pod \"payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp\" (UID: \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:46.677933 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.677892 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsw6l\" (UniqueName: \"kubernetes.io/projected/664423ef-4053-4db9-b4a4-85a93fb2e4fd-kube-api-access-gsw6l\") pod \"payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp\" (UID: \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:46.678097 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.678079 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664423ef-4053-4db9-b4a4-85a93fb2e4fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp\" (UID: \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:46.679520 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.679495 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664423ef-4053-4db9-b4a4-85a93fb2e4fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp\" (UID: \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:46.708640 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.708601 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch"] Nov 01 00:57:46.713750 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.713720 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-bf8ch"] Nov 01 00:57:46.741178 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.741099 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsw6l\" (UniqueName: \"kubernetes.io/projected/664423ef-4053-4db9-b4a4-85a93fb2e4fd-kube-api-access-gsw6l\") pod \"payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp\" (UID: \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:46.810741 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.810708 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:46.828971 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:46.828947 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:47.210299 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.210256 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp"] Nov 01 00:57:47.210841 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:57:47.210803 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664423ef_4053_4db9_b4a4_85a93fb2e4fd.slice/crio-7b6a41781a7e584a8a5cd044d42a7d4222012111bb716a7c1916da6b49a91b15 WatchSource:0}: Error finding container 7b6a41781a7e584a8a5cd044d42a7d4222012111bb716a7c1916da6b49a91b15: Status 404 returned error can't find the container with id 7b6a41781a7e584a8a5cd044d42a7d4222012111bb716a7c1916da6b49a91b15 Nov 01 00:57:47.253078 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.253044 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7cfb12-70c6-4469-9c8a-5997786d2125" path="/var/lib/kubelet/pods/9d7cfb12-70c6-4469-9c8a-5997786d2125/volumes" Nov 01 00:57:47.260665 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.260633 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh"] Nov 01 00:57:47.260914 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:57:47.260887 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dd805a_c7b1_49fb_9767_bc0309882f04.slice/crio-144a5a1f51584da619af522524cab0da85c078dbc9513b76de0748216c2ca437 WatchSource:0}: Error finding container 144a5a1f51584da619af522524cab0da85c078dbc9513b76de0748216c2ca437: Status 404 returned error can't find the container with id 144a5a1f51584da619af522524cab0da85c078dbc9513b76de0748216c2ca437 Nov 01 00:57:47.528754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.528625 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" event={"ID":"85dd805a-c7b1-49fb-9767-bc0309882f04","Type":"ContainerStarted","Data":"3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec"} Nov 01 00:57:47.528754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.528694 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" event={"ID":"85dd805a-c7b1-49fb-9767-bc0309882f04","Type":"ContainerStarted","Data":"144a5a1f51584da619af522524cab0da85c078dbc9513b76de0748216c2ca437"} Nov 01 00:57:47.528754 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.528725 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:47.530814 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.530782 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" event={"ID":"664423ef-4053-4db9-b4a4-85a93fb2e4fd","Type":"ContainerStarted","Data":"4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e"} Nov 01 00:57:47.530978 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.530835 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" event={"ID":"664423ef-4053-4db9-b4a4-85a93fb2e4fd","Type":"ContainerStarted","Data":"7b6a41781a7e584a8a5cd044d42a7d4222012111bb716a7c1916da6b49a91b15"} Nov 01 00:57:47.530978 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.530861 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:57:47.649890 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.649823 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" podStartSLOduration=1.649803133 podStartE2EDuration="1.649803133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:47.647149167 +0000 UTC m=+2012.522370186" watchObservedRunningTime="2026-11-01 00:57:47.649803133 +0000 UTC m=+2012.525024150" Nov 01 00:57:47.772312 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:47.770332 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" podStartSLOduration=1.770308881 podStartE2EDuration="1.770308881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:47.766884691 +0000 UTC m=+2012.642105726" watchObservedRunningTime="2026-11-01 00:57:47.770308881 +0000 UTC m=+2012.645529897" Nov 01 00:57:48.457648 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:48.457614 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 00:57:49.511523 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:57:49.511481 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:57:51.349554 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:51.349523 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:55.340669 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.340627 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:55.556070 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.555999 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:55.678508 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.678424 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d"] Nov 01 00:57:55.683341 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.683278 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:55.741051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.740994 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2919d823-3643-43f7-9a68-4640fdfc25c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d\" (UID: \"2919d823-3643-43f7-9a68-4640fdfc25c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:55.741268 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.741133 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vtn6\" (UniqueName: \"kubernetes.io/projected/2919d823-3643-43f7-9a68-4640fdfc25c8-kube-api-access-6vtn6\") pod \"payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d\" (UID: \"2919d823-3643-43f7-9a68-4640fdfc25c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:55.745858 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.744940 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d"] Nov 01 00:57:55.767130 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.767096 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:55.790608 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.790571 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-ba5b10\"" Nov 01 00:57:55.790608 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.790607 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-ba5b10-dockercfg-24x7f\"" Nov 01 00:57:55.843237 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.843195 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2919d823-3643-43f7-9a68-4640fdfc25c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d\" (UID: \"2919d823-3643-43f7-9a68-4640fdfc25c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:55.843465 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.843254 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vtn6\" (UniqueName: \"kubernetes.io/projected/2919d823-3643-43f7-9a68-4640fdfc25c8-kube-api-access-6vtn6\") pod \"payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d\" (UID: \"2919d823-3643-43f7-9a68-4640fdfc25c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:55.844066 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.844046 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2919d823-3643-43f7-9a68-4640fdfc25c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d\" (UID: \"2919d823-3643-43f7-9a68-4640fdfc25c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:55.867701 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.867662 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj"] Nov 01 00:57:55.874198 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.874163 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:55.913315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.912791 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vtn6\" (UniqueName: \"kubernetes.io/projected/2919d823-3643-43f7-9a68-4640fdfc25c8-kube-api-access-6vtn6\") pod \"payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d\" (UID: \"2919d823-3643-43f7-9a68-4640fdfc25c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:55.930887 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.930798 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj"] Nov 01 00:57:55.945299 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.945241 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcqt\" (UniqueName: \"kubernetes.io/projected/eac98067-e982-45a1-8b6a-022e2e331657-kube-api-access-nwcqt\") pod \"payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj\" (UID: \"eac98067-e982-45a1-8b6a-022e2e331657\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:55.945495 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:55.945321 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac98067-e982-45a1-8b6a-022e2e331657-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj\" (UID: \"eac98067-e982-45a1-8b6a-022e2e331657\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:56.016315 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.013589 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:56.050332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.047550 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcqt\" (UniqueName: \"kubernetes.io/projected/eac98067-e982-45a1-8b6a-022e2e331657-kube-api-access-nwcqt\") pod \"payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj\" (UID: \"eac98067-e982-45a1-8b6a-022e2e331657\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:56.050332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.047611 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac98067-e982-45a1-8b6a-022e2e331657-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj\" (UID: \"eac98067-e982-45a1-8b6a-022e2e331657\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:56.050332 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.048537 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac98067-e982-45a1-8b6a-022e2e331657-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj\" (UID: \"eac98067-e982-45a1-8b6a-022e2e331657\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:56.114904 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.114430 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcqt\" (UniqueName: \"kubernetes.io/projected/eac98067-e982-45a1-8b6a-022e2e331657-kube-api-access-nwcqt\") pod \"payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj\" (UID: \"eac98067-e982-45a1-8b6a-022e2e331657\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:56.204637 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.204523 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:56.461078 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:57:56.461044 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2919d823_3643_43f7_9a68_4640fdfc25c8.slice/crio-62923b1536152ee4542c4817ad187cf159feda0a471a6290270033142261da4f WatchSource:0}: Error finding container 62923b1536152ee4542c4817ad187cf159feda0a471a6290270033142261da4f: Status 404 returned error can't find the container with id 62923b1536152ee4542c4817ad187cf159feda0a471a6290270033142261da4f Nov 01 00:57:56.461480 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.461339 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d"] Nov 01 00:57:56.626594 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.608237 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj"] Nov 01 00:57:56.660094 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.660016 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" event={"ID":"2919d823-3643-43f7-9a68-4640fdfc25c8","Type":"ContainerStarted","Data":"62923b1536152ee4542c4817ad187cf159feda0a471a6290270033142261da4f"} Nov 01 00:57:56.662812 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:56.662756 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" event={"ID":"eac98067-e982-45a1-8b6a-022e2e331657","Type":"ContainerStarted","Data":"637d415517787e2e934bd964066db7ad989f413aa570c1bebdf66783005b1d17"} Nov 01 00:57:57.253638 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.253609 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:57.524962 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.524858 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:57.670758 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.670723 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" event={"ID":"2919d823-3643-43f7-9a68-4640fdfc25c8","Type":"ContainerStarted","Data":"f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90"} Nov 01 00:57:57.670967 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.670931 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:57:57.672590 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.672566 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" event={"ID":"eac98067-e982-45a1-8b6a-022e2e331657","Type":"ContainerStarted","Data":"b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915"} Nov 01 00:57:57.672717 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.672695 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:57:57.744124 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.744087 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:57.789810 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.789747 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" podStartSLOduration=2.789733502 podStartE2EDuration="2.789733502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:57.788758704 +0000 UTC m=+2022.663979748" watchObservedRunningTime="2026-11-01 00:57:57.789733502 +0000 UTC m=+2022.664954517" Nov 01 00:57:57.910952 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:57.910726 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" podStartSLOduration=2.9107041689999997 podStartE2EDuration="2.910704169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:57.90757604 +0000 UTC m=+2022.782797056" watchObservedRunningTime="2026-11-01 00:57:57.910704169 +0000 UTC m=+2022.785925185" Nov 01 00:57:58.178867 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:58.178780 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:57:58.543528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:58.543496 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 00:57:58.544014 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:57:58.543585 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 00:58:02.723897 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:02.723861 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:04.634321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:04.632437 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:04.863924 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:04.863844 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:05.086392 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:05.086071 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:05.997528 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:05.997491 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:06.632911 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:06.632823 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:07.034753 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:07.034713 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:07.658153 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:07.658108 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:08.470457 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:08.470416 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:58:08.470907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:08.470474 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:58:08.812347 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:08.812272 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 00:58:08.812530 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:08.812386 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 00:58:10.817541 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:10.817506 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:10.925321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:10.925270 6036 generic.go:356] "Generic (PLEG): container finished" podID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerID="dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8" exitCode=0 Nov 01 00:58:10.925512 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:10.925350 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" event={"ID":"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a","Type":"ContainerDied","Data":"dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8"} Nov 01 00:58:10.925907 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:10.925889 6036 scope.go:122] "RemoveContainer" containerID="dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8" Nov 01 00:58:10.927769 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:10.927736 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:58:10.927863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:10.927806 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:58:10.927863 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:10.927842 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:58:11.027934 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:11.027900 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:11.935898 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:11.935864 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" event={"ID":"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a","Type":"ContainerStarted","Data":"5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f"} Nov 01 00:58:12.812866 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:12.812828 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:58:12.813054 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:12.813025 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:58:12.813106 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:12.813076 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 00:58:13.128836 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:13.128756 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:13.343202 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:13.343170 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:13.812746 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:13.812708 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:58:13.812927 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:13.812764 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 00:58:14.813476 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:14.813435 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:58:14.813982 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:14.813501 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 00:58:15.812581 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:15.812542 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:58:15.812777 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:15.812612 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 00:58:16.141729 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:16.141612 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:16.382338 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:16.382246 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:16.812671 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:16.812630 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:58:16.812883 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:16.812678 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 00:58:17.817497 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:17.817465 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:58:17.817969 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:17.817935 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:58:17.818722 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:17.818702 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 00:58:18.187041 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:18.186957 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:19.117833 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:19.117794 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:19.506464 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:58:19.506379 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:58:20.411698 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:20.411660 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:21.008730 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:21.008684 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:23.029018 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:23.028982 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:27.835601 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:27.835567 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:32.082186 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:32.082146 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:32.627799 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:32.627760 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:37.834305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:37.832458 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-94qbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:58:37.834305 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:37.832538 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:58:37.856430 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:37.856389 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dd6kn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:58:37.856735 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:37.856700 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:58:38.095369 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:38.094830 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:42.978083 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:42.978037 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:43.187763 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:43.187729 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:43.406903 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:43.406870 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:48.058245 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:48.058212 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:49.506255 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:58:49.506217 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:58:53.372958 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:53.372920 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:53.590187 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:53.590148 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:57.624702 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:57.624670 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:58.126684 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:58.126650 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:58.546731 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:58.546699 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:58.794233 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:58.794196 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:58:59.268002 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:58:59.267960 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:03.827885 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:03.827854 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:05.994610 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:05.994578 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:07.610940 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:07.610908 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:08.002794 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:08.002707 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:08.667794 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:08.667702 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:09.081068 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:09.081033 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:09.290531 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:09.290499 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:09.514368 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:09.510141 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:13.998231 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:13.998197 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:14.251861 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:14.251750 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:14.463560 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:14.463522 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:14.684086 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:14.684052 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:17.405255 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:17.405181 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:17.634254 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:17.634218 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:17.964269 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:17.959672 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g"] Nov 01 00:59:18.014640 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.014609 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g"] Nov 01 00:59:18.014842 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.014731 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:18.041028 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.040991 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptds4\" (UniqueName: \"kubernetes.io/projected/4823c102-4b66-4381-a596-6e330e729e77-kube-api-access-ptds4\") pod \"payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g\" (UID: \"4823c102-4b66-4381-a596-6e330e729e77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:18.041229 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.041055 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4823c102-4b66-4381-a596-6e330e729e77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g\" (UID: \"4823c102-4b66-4381-a596-6e330e729e77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:18.069311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.068643 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-bc241b\"" Nov 01 00:59:18.120051 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.120009 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-bc241b-dockercfg-vfcpp\"" Nov 01 00:59:18.143172 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.143134 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptds4\" (UniqueName: \"kubernetes.io/projected/4823c102-4b66-4381-a596-6e330e729e77-kube-api-access-ptds4\") pod \"payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g\" (UID: \"4823c102-4b66-4381-a596-6e330e729e77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:18.143435 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.143186 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4823c102-4b66-4381-a596-6e330e729e77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g\" (UID: \"4823c102-4b66-4381-a596-6e330e729e77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:18.144062 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.144040 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4823c102-4b66-4381-a596-6e330e729e77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g\" (UID: \"4823c102-4b66-4381-a596-6e330e729e77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:18.210325 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.210245 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptds4\" (UniqueName: \"kubernetes.io/projected/4823c102-4b66-4381-a596-6e330e729e77-kube-api-access-ptds4\") pod \"payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g\" (UID: \"4823c102-4b66-4381-a596-6e330e729e77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:18.336337 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.336147 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:18.828765 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:18.828731 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g"] Nov 01 00:59:18.831767 ip-10-0-141-93 kubenswrapper[6036]: W1101 00:59:18.831719 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4823c102_4b66_4381_a596_6e330e729e77.slice/crio-691277955bdf3f9a7d41e2653e5e07d3824a1073724031551b893210d365b5f0 WatchSource:0}: Error finding container 691277955bdf3f9a7d41e2653e5e07d3824a1073724031551b893210d365b5f0: Status 404 returned error can't find the container with id 691277955bdf3f9a7d41e2653e5e07d3824a1073724031551b893210d365b5f0 Nov 01 00:59:19.526356 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:59:19.526321 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:59:19.623787 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:19.623749 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" event={"ID":"4823c102-4b66-4381-a596-6e330e729e77","Type":"ContainerStarted","Data":"c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5"} Nov 01 00:59:19.623981 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:19.623810 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" event={"ID":"4823c102-4b66-4381-a596-6e330e729e77","Type":"ContainerStarted","Data":"691277955bdf3f9a7d41e2653e5e07d3824a1073724031551b893210d365b5f0"} Nov 01 00:59:19.624050 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:19.623976 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:19.803407 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:19.803250 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" podStartSLOduration=2.803231371 podStartE2EDuration="2.803231371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:59:19.784915553 +0000 UTC m=+2104.660136568" watchObservedRunningTime="2026-11-01 00:59:19.803231371 +0000 UTC m=+2104.678452416" Nov 01 00:59:20.207435 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:20.200125 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:25.411442 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:25.411365 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:25.433821 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:25.433430 6036 patch_prober.go:29] interesting pod/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:59:25.433821 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:25.433515 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:59:25.732321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:25.732183 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:27.304328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.302735 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": read tcp 10.134.0.2:52856->10.134.0.99:15021: read: connection reset by peer" start-of-body= Nov 01 00:59:27.304328 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.302821 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": read tcp 10.134.0.2:52856->10.134.0.99:15021: read: connection reset by peer" Nov 01 00:59:27.716481 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.716440 6036 generic.go:356] "Generic (PLEG): container finished" podID="4f402208-c342-4429-8b0b-3259e410455d" containerID="b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea" exitCode=0 Nov 01 00:59:27.716734 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.716521 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" event={"ID":"4f402208-c342-4429-8b0b-3259e410455d","Type":"ContainerDied","Data":"b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea"} Nov 01 00:59:27.717105 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.717074 6036 scope.go:122] "RemoveContainer" containerID="b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea" Nov 01 00:59:27.718475 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.718449 6036 generic.go:356] "Generic (PLEG): container finished" podID="d8844a5a-492a-4534-a3af-07def8126146" containerID="d57fbafa75c687c35b4d1cf3e902bcc2ddc8189c75569512625a567fdcd317d8" exitCode=0 Nov 01 00:59:27.718636 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.718616 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" event={"ID":"d8844a5a-492a-4534-a3af-07def8126146","Type":"ContainerDied","Data":"d57fbafa75c687c35b4d1cf3e902bcc2ddc8189c75569512625a567fdcd317d8"} Nov 01 00:59:27.719013 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.718993 6036 scope.go:122] "RemoveContainer" containerID="d57fbafa75c687c35b4d1cf3e902bcc2ddc8189c75569512625a567fdcd317d8" Nov 01 00:59:27.719121 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.719031 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:27.719167 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.719116 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:27.719167 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.719151 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:27.720662 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.720634 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:27.720726 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.720696 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:27.720768 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:27.720728 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:28.727526 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:28.727493 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" event={"ID":"d8844a5a-492a-4534-a3af-07def8126146","Type":"ContainerStarted","Data":"56744e4edb65cc9e69c4feb326675cef32f08cc3d890faa600751d0b56023bc2"} Nov 01 00:59:28.729703 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:28.729677 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" event={"ID":"4f402208-c342-4429-8b0b-3259e410455d","Type":"ContainerStarted","Data":"9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54"} Nov 01 00:59:28.867112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:28.866660 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:59:28.867112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:28.866786 6036 patch_prober.go:29] interesting pod/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 00:59:28.867112 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:28.866830 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 00:59:29.502311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:29.501733 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:59:29.502311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:29.501870 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 00:59:29.502311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:29.501926 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 00:59:29.865313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:29.863745 6036 patch_prober.go:29] interesting pod/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 00:59:29.865313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:29.863811 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 00:59:30.500820 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:30.500693 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 00:59:30.500820 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:30.500756 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 00:59:30.667313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:30.643836 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 00:59:30.675320 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:30.671135 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:30.865323 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:30.864734 6036 patch_prober.go:29] interesting pod/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 00:59:30.865323 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:30.864800 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 00:59:31.500777 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:31.500738 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 00:59:31.501088 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:31.501055 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 00:59:31.868313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:31.864710 6036 patch_prober.go:29] interesting pod/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 00:59:31.868313 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:31.864773 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 00:59:32.497046 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:32.496756 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:32.499612 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:32.499578 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 00:59:32.499777 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:32.499643 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 00:59:32.866311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:32.865328 6036 patch_prober.go:29] interesting pod/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 00:59:32.866311 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:32.865394 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 00:59:33.499314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:33.499091 6036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 00:59:33.499314 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:33.499199 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 00:59:33.872321 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:33.863406 6036 patch_prober.go:29] interesting pod/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 00:59:33.892849 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:33.892807 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 00:59:34.505357 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:34.505324 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:59:34.505837 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:34.505820 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:59:34.506590 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:34.506572 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 00:59:34.866738 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:34.866710 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:59:34.867193 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:34.867174 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:59:34.868135 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:34.868112 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 00:59:35.550006 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:35.549953 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:40.356728 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:40.356695 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:43.367355 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:43.367317 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:43.586590 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:43.586558 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:45.330956 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:45.330922 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:46.302317 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:46.302245 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:49.507056 ip-10-0-141-93 kubenswrapper[6036]: E1101 00:59:49.507009 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 00:59:50.269891 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:50.269863 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:54.393923 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:54.393883 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:54.771698 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:54.771616 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 00:59:55.081744 ip-10-0-141-93 kubenswrapper[6036]: I1101 00:59:55.081706 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:00.038925 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.038888 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:00.194315 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.194268 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr"] Nov 01 01:00:00.199000 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.198971 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g"] Nov 01 01:00:00.199000 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.199000 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:00.203530 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.203507 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:00.231372 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.231349 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2"] Nov 01 01:00:00.235861 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.235842 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:00.253877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.253859 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:00:00.322352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.322233 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b9s9\" (UniqueName: \"kubernetes.io/projected/423c4873-4e43-436b-9de5-50c2829e4b42-kube-api-access-7b9s9\") pod \"maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr\" (UID: \"423c4873-4e43-436b-9de5-50c2829e4b42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:00.322512 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.322424 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxht9\" (UniqueName: \"kubernetes.io/projected/06def7de-075e-4a2d-81b4-c912ba9b5501-kube-api-access-bxht9\") pod \"maas-api-key-cleanup-29891580-xzzm2\" (UID: \"06def7de-075e-4a2d-81b4-c912ba9b5501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:00.322561 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.322511 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g\" (UID: \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:00.322807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.322785 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c4873-4e43-436b-9de5-50c2829e4b42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr\" (UID: \"423c4873-4e43-436b-9de5-50c2829e4b42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:00.322855 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.322839 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkn7b\" (UniqueName: \"kubernetes.io/projected/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-kube-api-access-pkn7b\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g\" (UID: \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:00.322955 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.322934 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06def7de-075e-4a2d-81b4-c912ba9b5501-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891580-xzzm2\" (UID: \"06def7de-075e-4a2d-81b4-c912ba9b5501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:00.386709 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.386663 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g"] Nov 01 01:00:00.392394 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.392349 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2"] Nov 01 01:00:00.424880 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.424841 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b9s9\" (UniqueName: \"kubernetes.io/projected/423c4873-4e43-436b-9de5-50c2829e4b42-kube-api-access-7b9s9\") pod \"maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr\" (UID: \"423c4873-4e43-436b-9de5-50c2829e4b42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:00.425034 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.424970 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxht9\" (UniqueName: \"kubernetes.io/projected/06def7de-075e-4a2d-81b4-c912ba9b5501-kube-api-access-bxht9\") pod \"maas-api-key-cleanup-29891580-xzzm2\" (UID: \"06def7de-075e-4a2d-81b4-c912ba9b5501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:00.425081 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.425038 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g\" (UID: \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:00.425257 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.425242 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c4873-4e43-436b-9de5-50c2829e4b42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr\" (UID: \"423c4873-4e43-436b-9de5-50c2829e4b42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:00.425375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.425305 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkn7b\" (UniqueName: \"kubernetes.io/projected/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-kube-api-access-pkn7b\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g\" (UID: \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:00.425375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.425360 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06def7de-075e-4a2d-81b4-c912ba9b5501-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891580-xzzm2\" (UID: \"06def7de-075e-4a2d-81b4-c912ba9b5501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:00.425933 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.425902 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g\" (UID: \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:00.426062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.425958 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c4873-4e43-436b-9de5-50c2829e4b42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr\" (UID: \"423c4873-4e43-436b-9de5-50c2829e4b42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:00.426062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.426032 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06def7de-075e-4a2d-81b4-c912ba9b5501-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891580-xzzm2\" (UID: \"06def7de-075e-4a2d-81b4-c912ba9b5501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:00.434941 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.434910 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr"] Nov 01 01:00:00.496929 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.489392 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b9s9\" (UniqueName: \"kubernetes.io/projected/423c4873-4e43-436b-9de5-50c2829e4b42-kube-api-access-7b9s9\") pod \"maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr\" (UID: \"423c4873-4e43-436b-9de5-50c2829e4b42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:00.496929 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.496828 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxht9\" (UniqueName: \"kubernetes.io/projected/06def7de-075e-4a2d-81b4-c912ba9b5501-kube-api-access-bxht9\") pod \"maas-api-key-cleanup-29891580-xzzm2\" (UID: \"06def7de-075e-4a2d-81b4-c912ba9b5501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:00.496929 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.496871 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkn7b\" (UniqueName: \"kubernetes.io/projected/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-kube-api-access-pkn7b\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g\" (UID: \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:00.535591 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.535560 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:00.545844 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.545823 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:00.554681 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.554660 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:00.937485 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.937410 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr"] Nov 01 01:00:00.946959 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:00:00.946930 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423c4873_4e43_436b_9de5_50c2829e4b42.slice/crio-a546f60b09be6d699a219d0cca9b3051c8a5b4cfeb1c20dbd4086839498dbf68 WatchSource:0}: Error finding container a546f60b09be6d699a219d0cca9b3051c8a5b4cfeb1c20dbd4086839498dbf68: Status 404 returned error can't find the container with id a546f60b09be6d699a219d0cca9b3051c8a5b4cfeb1c20dbd4086839498dbf68 Nov 01 01:00:00.985558 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:00:00.985526 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f561f46_f82e_44d5_b03c_7fa395ddb3f8.slice/crio-2d007c88695ca4415e4d6ffaf67e6795b9ffdf897170c0329fdf912b5b1a8847 WatchSource:0}: Error finding container 2d007c88695ca4415e4d6ffaf67e6795b9ffdf897170c0329fdf912b5b1a8847: Status 404 returned error can't find the container with id 2d007c88695ca4415e4d6ffaf67e6795b9ffdf897170c0329fdf912b5b1a8847 Nov 01 01:00:00.985711 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.985696 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g"] Nov 01 01:00:00.987948 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:00:00.987920 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06def7de_075e_4a2d_81b4_c912ba9b5501.slice/crio-9720191a3724517c744992c7c42adf5fb0e80f47c4c70180bed2b0f184519311 WatchSource:0}: Error finding container 9720191a3724517c744992c7c42adf5fb0e80f47c4c70180bed2b0f184519311: Status 404 returned error can't find the container with id 9720191a3724517c744992c7c42adf5fb0e80f47c4c70180bed2b0f184519311 Nov 01 01:00:00.989977 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:00.989953 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2"] Nov 01 01:00:01.044009 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:01.043978 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" event={"ID":"06def7de-075e-4a2d-81b4-c912ba9b5501","Type":"ContainerStarted","Data":"9720191a3724517c744992c7c42adf5fb0e80f47c4c70180bed2b0f184519311"} Nov 01 01:00:01.049780 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:01.049683 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" event={"ID":"423c4873-4e43-436b-9de5-50c2829e4b42","Type":"ContainerStarted","Data":"a546f60b09be6d699a219d0cca9b3051c8a5b4cfeb1c20dbd4086839498dbf68"} Nov 01 01:00:01.051542 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:01.051517 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" event={"ID":"3f561f46-f82e-44d5-b03c-7fa395ddb3f8","Type":"ContainerStarted","Data":"2d007c88695ca4415e4d6ffaf67e6795b9ffdf897170c0329fdf912b5b1a8847"} Nov 01 01:00:02.061690 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:02.061655 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" event={"ID":"06def7de-075e-4a2d-81b4-c912ba9b5501","Type":"ContainerStarted","Data":"f08e032c65b1ffdbfc3f52d11871313ba8a30c1b092e8f8ad5aed9714cb98f13"} Nov 01 01:00:02.063740 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:02.063709 6036 generic.go:356] "Generic (PLEG): container finished" podID="423c4873-4e43-436b-9de5-50c2829e4b42" containerID="778a7cef780dbc03e64e93447b679c53dc63b6072e2ae40e134aa08517d09631" exitCode=60 Nov 01 01:00:02.063866 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:02.063794 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" event={"ID":"423c4873-4e43-436b-9de5-50c2829e4b42","Type":"ContainerDied","Data":"778a7cef780dbc03e64e93447b679c53dc63b6072e2ae40e134aa08517d09631"} Nov 01 01:00:02.064125 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:02.064106 6036 scope.go:122] "RemoveContainer" containerID="778a7cef780dbc03e64e93447b679c53dc63b6072e2ae40e134aa08517d09631" Nov 01 01:00:02.065984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:02.065958 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" event={"ID":"3f561f46-f82e-44d5-b03c-7fa395ddb3f8","Type":"ContainerStarted","Data":"515b6ceee0502525f340e24abe1fcaaa26087ffe6d217cd975202519e04660cb"} Nov 01 01:00:02.184702 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:02.184644 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" podStartSLOduration=2.184628786 podStartE2EDuration="2.184628786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:02.1817004 +0000 UTC m=+2147.056921412" watchObservedRunningTime="2026-11-01 01:00:02.184628786 +0000 UTC m=+2147.059849801" Nov 01 01:00:02.417857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:02.417802 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" podStartSLOduration=2.417788892 podStartE2EDuration="2.417788892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:02.417066483 +0000 UTC m=+2147.292287498" watchObservedRunningTime="2026-11-01 01:00:02.417788892 +0000 UTC m=+2147.293009906" Nov 01 01:00:03.076086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:03.076055 6036 generic.go:356] "Generic (PLEG): container finished" podID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerID="f08e032c65b1ffdbfc3f52d11871313ba8a30c1b092e8f8ad5aed9714cb98f13" exitCode=60 Nov 01 01:00:03.076514 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:03.076143 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" event={"ID":"06def7de-075e-4a2d-81b4-c912ba9b5501","Type":"ContainerDied","Data":"f08e032c65b1ffdbfc3f52d11871313ba8a30c1b092e8f8ad5aed9714cb98f13"} Nov 01 01:00:03.076514 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:03.076473 6036 scope.go:122] "RemoveContainer" containerID="f08e032c65b1ffdbfc3f52d11871313ba8a30c1b092e8f8ad5aed9714cb98f13" Nov 01 01:00:03.078352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:03.078317 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" event={"ID":"423c4873-4e43-436b-9de5-50c2829e4b42","Type":"ContainerStarted","Data":"99f51b288d42ef04ceb5671ec61210ca1109cd0b9b17698168db961a873096d0"} Nov 01 01:00:03.080183 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:03.079927 6036 generic.go:356] "Generic (PLEG): container finished" podID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerID="515b6ceee0502525f340e24abe1fcaaa26087ffe6d217cd975202519e04660cb" exitCode=60 Nov 01 01:00:03.080183 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:03.079977 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" event={"ID":"3f561f46-f82e-44d5-b03c-7fa395ddb3f8","Type":"ContainerDied","Data":"515b6ceee0502525f340e24abe1fcaaa26087ffe6d217cd975202519e04660cb"} Nov 01 01:00:03.080339 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:03.080313 6036 scope.go:122] "RemoveContainer" containerID="515b6ceee0502525f340e24abe1fcaaa26087ffe6d217cd975202519e04660cb" Nov 01 01:00:03.318856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:03.318808 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" podStartSLOduration=3.318794422 podStartE2EDuration="3.318794422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:03.316099483 +0000 UTC m=+2148.191320499" watchObservedRunningTime="2026-11-01 01:00:03.318794422 +0000 UTC m=+2148.194015436" Nov 01 01:00:04.089608 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.089578 6036 generic.go:356] "Generic (PLEG): container finished" podID="423c4873-4e43-436b-9de5-50c2829e4b42" containerID="99f51b288d42ef04ceb5671ec61210ca1109cd0b9b17698168db961a873096d0" exitCode=60 Nov 01 01:00:04.089990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.089665 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" event={"ID":"423c4873-4e43-436b-9de5-50c2829e4b42","Type":"ContainerDied","Data":"99f51b288d42ef04ceb5671ec61210ca1109cd0b9b17698168db961a873096d0"} Nov 01 01:00:04.089990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.089768 6036 scope.go:122] "RemoveContainer" containerID="778a7cef780dbc03e64e93447b679c53dc63b6072e2ae40e134aa08517d09631" Nov 01 01:00:04.089990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.089966 6036 scope.go:122] "RemoveContainer" containerID="99f51b288d42ef04ceb5671ec61210ca1109cd0b9b17698168db961a873096d0" Nov 01 01:00:04.090398 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:04.090375 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr_odh-ai-gateway-infra(423c4873-4e43-436b-9de5-50c2829e4b42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" Nov 01 01:00:04.091896 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.091862 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" event={"ID":"3f561f46-f82e-44d5-b03c-7fa395ddb3f8","Type":"ContainerStarted","Data":"a86de61b7b3f66468a3e0b8192f5d1e18a473acbc16b02c590c309bbfabb99ca"} Nov 01 01:00:04.093807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.093786 6036 generic.go:356] "Generic (PLEG): container finished" podID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerID="400e325127d78579595bf8bca6dff557e8c81cf4363695d1724eda51ae34ddc4" exitCode=60 Nov 01 01:00:04.093895 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.093830 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" event={"ID":"06def7de-075e-4a2d-81b4-c912ba9b5501","Type":"ContainerDied","Data":"400e325127d78579595bf8bca6dff557e8c81cf4363695d1724eda51ae34ddc4"} Nov 01 01:00:04.094060 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.094044 6036 scope.go:122] "RemoveContainer" containerID="400e325127d78579595bf8bca6dff557e8c81cf4363695d1724eda51ae34ddc4" Nov 01 01:00:04.094309 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:04.094267 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891580-xzzm2_odh-ai-gateway-infra(06def7de-075e-4a2d-81b4-c912ba9b5501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" Nov 01 01:00:04.125001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:04.124980 6036 scope.go:122] "RemoveContainer" containerID="f08e032c65b1ffdbfc3f52d11871313ba8a30c1b092e8f8ad5aed9714cb98f13" Nov 01 01:00:05.103051 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:05.103016 6036 scope.go:122] "RemoveContainer" containerID="400e325127d78579595bf8bca6dff557e8c81cf4363695d1724eda51ae34ddc4" Nov 01 01:00:05.103554 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:05.103453 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891580-xzzm2_odh-ai-gateway-infra(06def7de-075e-4a2d-81b4-c912ba9b5501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" Nov 01 01:00:05.104784 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:05.104760 6036 scope.go:122] "RemoveContainer" containerID="99f51b288d42ef04ceb5671ec61210ca1109cd0b9b17698168db961a873096d0" Nov 01 01:00:05.105117 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:05.105097 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr_odh-ai-gateway-infra(423c4873-4e43-436b-9de5-50c2829e4b42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" Nov 01 01:00:05.106405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:05.106376 6036 generic.go:356] "Generic (PLEG): container finished" podID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerID="a86de61b7b3f66468a3e0b8192f5d1e18a473acbc16b02c590c309bbfabb99ca" exitCode=60 Nov 01 01:00:05.106501 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:05.106449 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" event={"ID":"3f561f46-f82e-44d5-b03c-7fa395ddb3f8","Type":"ContainerDied","Data":"a86de61b7b3f66468a3e0b8192f5d1e18a473acbc16b02c590c309bbfabb99ca"} Nov 01 01:00:05.106551 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:05.106518 6036 scope.go:122] "RemoveContainer" containerID="515b6ceee0502525f340e24abe1fcaaa26087ffe6d217cd975202519e04660cb" Nov 01 01:00:05.106728 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:05.106715 6036 scope.go:122] "RemoveContainer" containerID="a86de61b7b3f66468a3e0b8192f5d1e18a473acbc16b02c590c309bbfabb99ca" Nov 01 01:00:05.107093 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:05.107060 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g_odh-ai-gateway-infra(3f561f46-f82e-44d5-b03c-7fa395ddb3f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" Nov 01 01:00:05.390007 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:05.389919 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:06.115780 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:06.115750 6036 scope.go:122] "RemoveContainer" containerID="a86de61b7b3f66468a3e0b8192f5d1e18a473acbc16b02c590c309bbfabb99ca" Nov 01 01:00:06.116180 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:06.116094 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g_odh-ai-gateway-infra(3f561f46-f82e-44d5-b03c-7fa395ddb3f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" Nov 01 01:00:09.886979 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:09.886940 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:10.699201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:10.699163 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:11.120133 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:11.120093 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:12.359527 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:12.359452 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:12.587016 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:12.586972 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:14.243758 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:14.243723 6036 scope.go:122] "RemoveContainer" containerID="99f51b288d42ef04ceb5671ec61210ca1109cd0b9b17698168db961a873096d0" Nov 01 01:00:15.200347 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:15.200309 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" event={"ID":"423c4873-4e43-436b-9de5-50c2829e4b42","Type":"ContainerStarted","Data":"cb774e2bfaac2b8037d30eeb8395723827259d2acda1797b6cc7d9466a4b13e9"} Nov 01 01:00:16.072419 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:16.072381 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:16.209747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:16.209710 6036 generic.go:356] "Generic (PLEG): container finished" podID="423c4873-4e43-436b-9de5-50c2829e4b42" containerID="cb774e2bfaac2b8037d30eeb8395723827259d2acda1797b6cc7d9466a4b13e9" exitCode=60 Nov 01 01:00:16.209928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:16.209779 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" event={"ID":"423c4873-4e43-436b-9de5-50c2829e4b42","Type":"ContainerDied","Data":"cb774e2bfaac2b8037d30eeb8395723827259d2acda1797b6cc7d9466a4b13e9"} Nov 01 01:00:16.209928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:16.209866 6036 scope.go:122] "RemoveContainer" containerID="99f51b288d42ef04ceb5671ec61210ca1109cd0b9b17698168db961a873096d0" Nov 01 01:00:16.210266 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:16.210247 6036 scope.go:122] "RemoveContainer" containerID="cb774e2bfaac2b8037d30eeb8395723827259d2acda1797b6cc7d9466a4b13e9" Nov 01 01:00:16.210689 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:16.210669 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr_odh-ai-gateway-infra(423c4873-4e43-436b-9de5-50c2829e4b42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" Nov 01 01:00:16.244583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:16.244552 6036 scope.go:122] "RemoveContainer" containerID="a86de61b7b3f66468a3e0b8192f5d1e18a473acbc16b02c590c309bbfabb99ca" Nov 01 01:00:16.285506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:16.285481 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:16.332465 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:16.332388 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr"] Nov 01 01:00:17.221592 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.221562 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" event={"ID":"3f561f46-f82e-44d5-b03c-7fa395ddb3f8","Type":"ContainerStarted","Data":"d81de8bf1e1b67db78c19f92a6e73d2d93376e880754a4b5706de083ef300c93"} Nov 01 01:00:17.389546 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.389520 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:17.525367 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.525246 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423c4873-4e43-436b-9de5-50c2829e4b42-kube-api-access-7b9s9\" (UniqueName: \"kubernetes.io/projected/423c4873-4e43-436b-9de5-50c2829e4b42-kube-api-access-7b9s9\") pod \"423c4873-4e43-436b-9de5-50c2829e4b42\" (UID: \"423c4873-4e43-436b-9de5-50c2829e4b42\") " Nov 01 01:00:17.525367 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.525354 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423c4873-4e43-436b-9de5-50c2829e4b42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c4873-4e43-436b-9de5-50c2829e4b42-maas-api-service-ca\") pod \"423c4873-4e43-436b-9de5-50c2829e4b42\" (UID: \"423c4873-4e43-436b-9de5-50c2829e4b42\") " Nov 01 01:00:17.525779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.525751 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423c4873-4e43-436b-9de5-50c2829e4b42-maas-api-service-ca" pod "423c4873-4e43-436b-9de5-50c2829e4b42" (UID: "423c4873-4e43-436b-9de5-50c2829e4b42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:17.527718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.527678 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423c4873-4e43-436b-9de5-50c2829e4b42-kube-api-access-7b9s9" pod "423c4873-4e43-436b-9de5-50c2829e4b42" (UID: "423c4873-4e43-436b-9de5-50c2829e4b42"). InnerVolumeSpecName "kube-api-access-7b9s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:17.548262 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.548233 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:17.627333 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.627271 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c4873-4e43-436b-9de5-50c2829e4b42-maas-api-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:17.627333 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.627331 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b9s9\" (UniqueName: \"kubernetes.io/projected/423c4873-4e43-436b-9de5-50c2829e4b42-kube-api-access-7b9s9\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:17.769068 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:17.769037 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:18.006070 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.006033 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:18.230912 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.230888 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" Nov 01 01:00:18.230912 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.230892 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr" event={"ID":"423c4873-4e43-436b-9de5-50c2829e4b42","Type":"ContainerDied","Data":"a546f60b09be6d699a219d0cca9b3051c8a5b4cfeb1c20dbd4086839498dbf68"} Nov 01 01:00:18.231509 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.230957 6036 scope.go:122] "RemoveContainer" containerID="cb774e2bfaac2b8037d30eeb8395723827259d2acda1797b6cc7d9466a4b13e9" Nov 01 01:00:18.233193 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.233171 6036 generic.go:356] "Generic (PLEG): container finished" podID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerID="d81de8bf1e1b67db78c19f92a6e73d2d93376e880754a4b5706de083ef300c93" exitCode=60 Nov 01 01:00:18.233356 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.233251 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" event={"ID":"3f561f46-f82e-44d5-b03c-7fa395ddb3f8","Type":"ContainerDied","Data":"d81de8bf1e1b67db78c19f92a6e73d2d93376e880754a4b5706de083ef300c93"} Nov 01 01:00:18.233852 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.233832 6036 scope.go:122] "RemoveContainer" containerID="d81de8bf1e1b67db78c19f92a6e73d2d93376e880754a4b5706de083ef300c93" Nov 01 01:00:18.234246 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:18.234224 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g_odh-ai-gateway-infra(3f561f46-f82e-44d5-b03c-7fa395ddb3f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" Nov 01 01:00:18.242528 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.242504 6036 scope.go:122] "RemoveContainer" containerID="a86de61b7b3f66468a3e0b8192f5d1e18a473acbc16b02c590c309bbfabb99ca" Nov 01 01:00:18.242991 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.242971 6036 scope.go:122] "RemoveContainer" containerID="400e325127d78579595bf8bca6dff557e8c81cf4363695d1724eda51ae34ddc4" Nov 01 01:00:18.469314 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.469247 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g"] Nov 01 01:00:18.533770 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.533742 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr"] Nov 01 01:00:18.536960 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:18.536933 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f97e89-29891580-swkgr"] Nov 01 01:00:19.243004 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.242963 6036 generic.go:356] "Generic (PLEG): container finished" podID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerID="4443ddc5815f59a1f912706fdda11c0615da2c97787917c3eb30def9dc0be5fa" exitCode=60 Nov 01 01:00:19.254528 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.254496 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" path="/var/lib/kubelet/pods/423c4873-4e43-436b-9de5-50c2829e4b42/volumes" Nov 01 01:00:19.255042 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.255011 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" event={"ID":"06def7de-075e-4a2d-81b4-c912ba9b5501","Type":"ContainerDied","Data":"4443ddc5815f59a1f912706fdda11c0615da2c97787917c3eb30def9dc0be5fa"} Nov 01 01:00:19.255122 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.255096 6036 scope.go:122] "RemoveContainer" containerID="400e325127d78579595bf8bca6dff557e8c81cf4363695d1724eda51ae34ddc4" Nov 01 01:00:19.255552 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.255530 6036 scope.go:122] "RemoveContainer" containerID="4443ddc5815f59a1f912706fdda11c0615da2c97787917c3eb30def9dc0be5fa" Nov 01 01:00:19.255967 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:19.255938 6036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891580-xzzm2_odh-ai-gateway-infra(06def7de-075e-4a2d-81b4-c912ba9b5501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" Nov 01 01:00:19.429342 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.429315 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:19.510370 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:19.510335 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:00:19.561059 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.561028 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-kube-api-access-pkn7b\" (UniqueName: \"kubernetes.io/projected/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-kube-api-access-pkn7b\") pod \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\" (UID: \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\") " Nov 01 01:00:19.561339 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.561324 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-maas-api-service-ca\") pod \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\" (UID: \"3f561f46-f82e-44d5-b03c-7fa395ddb3f8\") " Nov 01 01:00:19.561680 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.561655 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-maas-api-service-ca" pod "3f561f46-f82e-44d5-b03c-7fa395ddb3f8" (UID: "3f561f46-f82e-44d5-b03c-7fa395ddb3f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:19.561844 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.561827 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-maas-api-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:19.563270 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.563246 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-kube-api-access-pkn7b" pod "3f561f46-f82e-44d5-b03c-7fa395ddb3f8" (UID: "3f561f46-f82e-44d5-b03c-7fa395ddb3f8"). InnerVolumeSpecName "kube-api-access-pkn7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:19.664012 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:19.663929 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkn7b\" (UniqueName: \"kubernetes.io/projected/3f561f46-f82e-44d5-b03c-7fa395ddb3f8-kube-api-access-pkn7b\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.254227 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.254199 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" Nov 01 01:00:20.254757 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.254235 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g" event={"ID":"3f561f46-f82e-44d5-b03c-7fa395ddb3f8","Type":"ContainerDied","Data":"2d007c88695ca4415e4d6ffaf67e6795b9ffdf897170c0329fdf912b5b1a8847"} Nov 01 01:00:20.254757 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.254328 6036 scope.go:122] "RemoveContainer" containerID="d81de8bf1e1b67db78c19f92a6e73d2d93376e880754a4b5706de083ef300c93" Nov 01 01:00:20.457581 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.457546 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g"] Nov 01 01:00:20.466837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.466808 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891580-pbx5g"] Nov 01 01:00:20.509042 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.508969 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2"] Nov 01 01:00:20.681371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.681346 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:20.780402 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.780276 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06def7de-075e-4a2d-81b4-c912ba9b5501-kube-api-access-bxht9\" (UniqueName: \"kubernetes.io/projected/06def7de-075e-4a2d-81b4-c912ba9b5501-kube-api-access-bxht9\") pod \"06def7de-075e-4a2d-81b4-c912ba9b5501\" (UID: \"06def7de-075e-4a2d-81b4-c912ba9b5501\") " Nov 01 01:00:20.780598 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.780575 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06def7de-075e-4a2d-81b4-c912ba9b5501-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06def7de-075e-4a2d-81b4-c912ba9b5501-maas-api-service-ca\") pod \"06def7de-075e-4a2d-81b4-c912ba9b5501\" (UID: \"06def7de-075e-4a2d-81b4-c912ba9b5501\") " Nov 01 01:00:20.780948 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.780922 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06def7de-075e-4a2d-81b4-c912ba9b5501-maas-api-service-ca" pod "06def7de-075e-4a2d-81b4-c912ba9b5501" (UID: "06def7de-075e-4a2d-81b4-c912ba9b5501"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:20.781202 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.781183 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06def7de-075e-4a2d-81b4-c912ba9b5501-maas-api-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.783065 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.783044 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06def7de-075e-4a2d-81b4-c912ba9b5501-kube-api-access-bxht9" pod "06def7de-075e-4a2d-81b4-c912ba9b5501" (UID: "06def7de-075e-4a2d-81b4-c912ba9b5501"). InnerVolumeSpecName "kube-api-access-bxht9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:20.875982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.875944 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:20.883712 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:20.883648 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxht9\" (UniqueName: \"kubernetes.io/projected/06def7de-075e-4a2d-81b4-c912ba9b5501-kube-api-access-bxht9\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:21.255639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.255606 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:21.260557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.260520 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" path="/var/lib/kubelet/pods/3f561f46-f82e-44d5-b03c-7fa395ddb3f8/volumes" Nov 01 01:00:21.275278 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.275254 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" Nov 01 01:00:21.275399 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.275275 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2" event={"ID":"06def7de-075e-4a2d-81b4-c912ba9b5501","Type":"ContainerDied","Data":"9720191a3724517c744992c7c42adf5fb0e80f47c4c70180bed2b0f184519311"} Nov 01 01:00:21.275501 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.275401 6036 scope.go:122] "RemoveContainer" containerID="4443ddc5815f59a1f912706fdda11c0615da2c97787917c3eb30def9dc0be5fa" Nov 01 01:00:21.461954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.461889 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2"] Nov 01 01:00:21.472193 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.467235 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891580-xzzm2"] Nov 01 01:00:21.497314 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.486488 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:21.716982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.716950 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:21.717958 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.717937 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54"] Nov 01 01:00:21.718654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718640 6036 cpu_manager.go:414] "containerMap: removing container" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:21.718715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718655 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:21.718715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718672 6036 cpu_manager.go:414] "containerMap: removing container" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.718715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718677 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.718715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718689 6036 cpu_manager.go:414] "containerMap: removing container" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:21.718715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718694 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:21.718715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718703 6036 cpu_manager.go:414] "containerMap: removing container" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:21.718715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718707 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718723 6036 cpu_manager.go:414] "containerMap: removing container" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718728 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718739 6036 cpu_manager.go:414] "containerMap: removing container" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718743 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718755 6036 cpu_manager.go:414] "containerMap: removing container" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718760 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718767 6036 cpu_manager.go:414] "containerMap: removing container" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718771 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718878 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718888 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718896 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718902 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718908 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718917 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718924 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="423c4873-4e43-436b-9de5-50c2829e4b42" containerName="cleanup" Nov 01 01:00:21.719062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.718937 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.723587 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.723568 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:21.780838 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.780706 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0f9d81\"" Nov 01 01:00:21.781240 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.781219 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0f9d81-dockercfg-22zsb\"" Nov 01 01:00:21.781438 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.781361 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54"] Nov 01 01:00:21.816777 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.816140 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599bb32b-a4fd-4cb6-9787-4ae7885dc770-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54\" (UID: \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:21.816777 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.816201 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qf6r\" (UniqueName: \"kubernetes.io/projected/599bb32b-a4fd-4cb6-9787-4ae7885dc770-kube-api-access-5qf6r\") pod \"payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54\" (UID: \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:21.944398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.942793 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599bb32b-a4fd-4cb6-9787-4ae7885dc770-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54\" (UID: \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:21.944398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.942862 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qf6r\" (UniqueName: \"kubernetes.io/projected/599bb32b-a4fd-4cb6-9787-4ae7885dc770-kube-api-access-5qf6r\") pod \"payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54\" (UID: \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:21.944651 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.944411 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599bb32b-a4fd-4cb6-9787-4ae7885dc770-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54\" (UID: \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:21.977364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.976363 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb"] Nov 01 01:00:21.979747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.978412 6036 cpu_manager.go:414] "containerMap: removing container" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.979747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.978440 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f561f46-f82e-44d5-b03c-7fa395ddb3f8" containerName="cleanup" Nov 01 01:00:21.979747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:21.978888 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" containerName="cleanup" Nov 01 01:00:22.011234 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.007698 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb"] Nov 01 01:00:22.011234 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.007882 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:22.025331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.023261 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qf6r\" (UniqueName: \"kubernetes.io/projected/599bb32b-a4fd-4cb6-9787-4ae7885dc770-kube-api-access-5qf6r\") pod \"payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54\" (UID: \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:22.051095 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.051059 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:22.053161 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.052991 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb\" (UID: \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:22.053161 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.053076 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdjx\" (UniqueName: \"kubernetes.io/projected/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-kube-api-access-cqdjx\") pod \"payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb\" (UID: \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:22.156591 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.156547 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb\" (UID: \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:22.156775 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.156649 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjx\" (UniqueName: \"kubernetes.io/projected/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-kube-api-access-cqdjx\") pod \"payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb\" (UID: \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:22.158241 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.158207 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb\" (UID: \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:22.159656 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.159622 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:22.238872 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.236040 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdjx\" (UniqueName: \"kubernetes.io/projected/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-kube-api-access-cqdjx\") pod \"payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb\" (UID: \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:22.381070 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.381036 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:22.474169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.474139 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54"] Nov 01 01:00:22.478018 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:00:22.477990 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599bb32b_a4fd_4cb6_9787_4ae7885dc770.slice/crio-e1fe682b91ab88c751c1365719f98f69b9f1f8239213a7e8fca79db0f90952cc WatchSource:0}: Error finding container e1fe682b91ab88c751c1365719f98f69b9f1f8239213a7e8fca79db0f90952cc: Status 404 returned error can't find the container with id e1fe682b91ab88c751c1365719f98f69b9f1f8239213a7e8fca79db0f90952cc Nov 01 01:00:22.628480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.628444 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:22.773480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:22.773453 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb"] Nov 01 01:00:22.775715 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:00:22.775684 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee7203e_1654_44fc_9f9a_5c3d5d7db131.slice/crio-35e86a73add1090d001b498cb1e014be40331283e9abab36ad1073c6015f5d46 WatchSource:0}: Error finding container 35e86a73add1090d001b498cb1e014be40331283e9abab36ad1073c6015f5d46: Status 404 returned error can't find the container with id 35e86a73add1090d001b498cb1e014be40331283e9abab36ad1073c6015f5d46 Nov 01 01:00:23.253324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.253254 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06def7de-075e-4a2d-81b4-c912ba9b5501" path="/var/lib/kubelet/pods/06def7de-075e-4a2d-81b4-c912ba9b5501/volumes" Nov 01 01:00:23.299368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.299328 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" event={"ID":"8ee7203e-1654-44fc-9f9a-5c3d5d7db131","Type":"ContainerStarted","Data":"86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba"} Nov 01 01:00:23.299572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.299392 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" event={"ID":"8ee7203e-1654-44fc-9f9a-5c3d5d7db131","Type":"ContainerStarted","Data":"35e86a73add1090d001b498cb1e014be40331283e9abab36ad1073c6015f5d46"} Nov 01 01:00:23.299572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.299453 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:23.301311 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.301232 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" event={"ID":"599bb32b-a4fd-4cb6-9787-4ae7885dc770","Type":"ContainerStarted","Data":"cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640"} Nov 01 01:00:23.301433 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.301313 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" event={"ID":"599bb32b-a4fd-4cb6-9787-4ae7885dc770","Type":"ContainerStarted","Data":"e1fe682b91ab88c751c1365719f98f69b9f1f8239213a7e8fca79db0f90952cc"} Nov 01 01:00:23.301433 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.301344 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:23.452880 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.452823 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" podStartSLOduration=2.452808827 podStartE2EDuration="2.452808827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:23.451762556 +0000 UTC m=+2168.326983572" watchObservedRunningTime="2026-11-01 01:00:23.452808827 +0000 UTC m=+2168.328029842" Nov 01 01:00:23.576979 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:23.576929 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" podStartSLOduration=2.576915142 podStartE2EDuration="2.576915142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:23.576357658 +0000 UTC m=+2168.451578674" watchObservedRunningTime="2026-11-01 01:00:23.576915142 +0000 UTC m=+2168.452136157" Nov 01 01:00:24.095477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:24.095426 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:27.135232 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:27.135192 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:27.426041 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:27.425936 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:27.635480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:27.635441 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:31.060620 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.060578 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:31.286145 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.286041 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:31.404497 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.404416 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f"] Nov 01 01:00:31.410490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.410461 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:31.468873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.468837 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f"] Nov 01 01:00:31.540496 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.517685 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-de2a95-dockercfg-m6nkp\"" Nov 01 01:00:31.540496 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.517809 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-de2a95\"" Nov 01 01:00:31.575961 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.543640 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f\" (UID: \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:31.576857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.576815 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxxc\" (UniqueName: \"kubernetes.io/projected/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-kube-api-access-nmxxc\") pod \"payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f\" (UID: \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:31.660414 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.653317 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4"] Nov 01 01:00:31.674806 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.674764 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4"] Nov 01 01:00:31.674982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.674921 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:31.678723 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.678683 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f\" (UID: \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:31.678924 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.678901 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxxc\" (UniqueName: \"kubernetes.io/projected/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-kube-api-access-nmxxc\") pod \"payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f\" (UID: \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:31.679953 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.679925 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f\" (UID: \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:31.764899 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.764864 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxxc\" (UniqueName: \"kubernetes.io/projected/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-kube-api-access-nmxxc\") pod \"payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f\" (UID: \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:31.780194 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.780158 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4\" (UID: \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:31.780431 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.780410 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjzk\" (UniqueName: \"kubernetes.io/projected/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-kube-api-access-mbjzk\") pod \"payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4\" (UID: \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:31.883910 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.883866 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4\" (UID: \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:31.884094 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.884035 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjzk\" (UniqueName: \"kubernetes.io/projected/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-kube-api-access-mbjzk\") pod \"payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4\" (UID: \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:31.884818 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.884784 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4\" (UID: \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:31.947395 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.947320 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjzk\" (UniqueName: \"kubernetes.io/projected/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-kube-api-access-mbjzk\") pod \"payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4\" (UID: \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:31.999416 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:31.999380 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:32.028946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:32.028912 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:32.393298 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:32.393263 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4"] Nov 01 01:00:32.431556 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:00:32.431528 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fb71c9_8b5e_4b46_bcdb_b875d3f0c2b1.slice/crio-0801cc7a8f024c6adbaf989cc6f3809686ee0d298b276fc729b55f6b4f561962 WatchSource:0}: Error finding container 0801cc7a8f024c6adbaf989cc6f3809686ee0d298b276fc729b55f6b4f561962: Status 404 returned error can't find the container with id 0801cc7a8f024c6adbaf989cc6f3809686ee0d298b276fc729b55f6b4f561962 Nov 01 01:00:32.448133 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:00:32.448104 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712175d1_b5eb_449f_baaa_8ffdd8c7fe4f.slice/crio-4e7c3a4b07d201672cb354806d08f5376eb5eaffa94e24ea7333a81926b3c739 WatchSource:0}: Error finding container 4e7c3a4b07d201672cb354806d08f5376eb5eaffa94e24ea7333a81926b3c739: Status 404 returned error can't find the container with id 4e7c3a4b07d201672cb354806d08f5376eb5eaffa94e24ea7333a81926b3c739 Nov 01 01:00:32.449954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:32.449930 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f"] Nov 01 01:00:32.586277 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:32.586238 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:32.863303 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:32.863235 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:33.379936 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:33.379892 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" event={"ID":"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1","Type":"ContainerStarted","Data":"2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532"} Nov 01 01:00:33.380122 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:33.379996 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" event={"ID":"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1","Type":"ContainerStarted","Data":"0801cc7a8f024c6adbaf989cc6f3809686ee0d298b276fc729b55f6b4f561962"} Nov 01 01:00:33.380231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:33.380210 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:33.381653 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:33.381626 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" event={"ID":"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f","Type":"ContainerStarted","Data":"0826badebb08010ed764eaf2b1e3d239013ab08822e6537b9dc0bb764d24768e"} Nov 01 01:00:33.381787 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:33.381665 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" event={"ID":"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f","Type":"ContainerStarted","Data":"4e7c3a4b07d201672cb354806d08f5376eb5eaffa94e24ea7333a81926b3c739"} Nov 01 01:00:33.381787 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:33.381723 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:33.499275 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:33.499226 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" podStartSLOduration=2.499212598 podStartE2EDuration="2.499212598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:33.497798151 +0000 UTC m=+2178.373019165" watchObservedRunningTime="2026-11-01 01:00:33.499212598 +0000 UTC m=+2178.374433612" Nov 01 01:00:33.620535 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:33.620471 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" podStartSLOduration=2.620450264 podStartE2EDuration="2.620450264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:33.617511378 +0000 UTC m=+2178.492732397" watchObservedRunningTime="2026-11-01 01:00:33.620450264 +0000 UTC m=+2178.495671280" Nov 01 01:00:34.312918 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:34.312888 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:00:34.313100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:34.312964 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:00:37.294660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:37.294602 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:37.509850 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:37.509821 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:37.791363 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:37.791316 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:42.368758 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:42.368714 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:44.391968 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:44.391934 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:00:44.392473 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:44.392013 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:00:47.338761 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:47.338682 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:49.506859 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:00:49.506819 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:00:52.326943 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:52.326911 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:55.837074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:55.837040 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:57.357455 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:57.354130 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:00:59.780626 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:00:59.780587 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:00.563165 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:00.563131 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:01.218766 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:01.218728 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:01.687992 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:01.687959 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:03.193724 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:03.193692 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:05.689355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:05.689317 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:05.901572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:05.901524 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:08.475673 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:08.475637 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:09.456364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.456332 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:09.675849 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.675716 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:09.814731 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.814698 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f"] Nov 01 01:01:09.820179 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.820153 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:09.877129 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.877092 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f"] Nov 01 01:01:09.919626 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.919589 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmns\" (UniqueName: \"kubernetes.io/projected/092dea27-ada1-4838-ab3b-a38b4dbc4d97-kube-api-access-tqmns\") pod \"payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f\" (UID: \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:09.920414 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.920381 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092dea27-ada1-4838-ab3b-a38b4dbc4d97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f\" (UID: \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:09.923981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.923958 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f73fb0-dockercfg-tdv4s\"" Nov 01 01:01:09.925606 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:09.925543 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f73fb0\"" Nov 01 01:01:10.022480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.022444 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmns\" (UniqueName: \"kubernetes.io/projected/092dea27-ada1-4838-ab3b-a38b4dbc4d97-kube-api-access-tqmns\") pod \"payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f\" (UID: \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:10.023018 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.022993 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092dea27-ada1-4838-ab3b-a38b4dbc4d97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f\" (UID: \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:10.024123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.024098 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092dea27-ada1-4838-ab3b-a38b4dbc4d97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f\" (UID: \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:10.025773 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.025437 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf"] Nov 01 01:01:10.032367 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.031881 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:10.110367 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.102542 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmns\" (UniqueName: \"kubernetes.io/projected/092dea27-ada1-4838-ab3b-a38b4dbc4d97-kube-api-access-tqmns\") pod \"payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f\" (UID: \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:10.120063 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.114657 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf"] Nov 01 01:01:10.130219 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.129833 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf768\" (UniqueName: \"kubernetes.io/projected/9c47c543-f8cd-4a7c-811f-2f884e97a10b-kube-api-access-vf768\") pod \"payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf\" (UID: \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:10.130219 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.130055 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47c543-f8cd-4a7c-811f-2f884e97a10b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf\" (UID: \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:10.142566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.140118 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:10.236347 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.233450 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47c543-f8cd-4a7c-811f-2f884e97a10b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf\" (UID: \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:10.236347 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.233616 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf768\" (UniqueName: \"kubernetes.io/projected/9c47c543-f8cd-4a7c-811f-2f884e97a10b-kube-api-access-vf768\") pod \"payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf\" (UID: \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:10.236347 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.235517 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47c543-f8cd-4a7c-811f-2f884e97a10b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf\" (UID: \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:10.299398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.299363 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf768\" (UniqueName: \"kubernetes.io/projected/9c47c543-f8cd-4a7c-811f-2f884e97a10b-kube-api-access-vf768\") pod \"payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf\" (UID: \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:10.358944 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.358903 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:10.536791 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.536765 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f"] Nov 01 01:01:10.537153 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:01:10.537123 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092dea27_ada1_4838_ab3b_a38b4dbc4d97.slice/crio-66c9ee410ad682a8c546691b1bfe13ea7f49ed7546b175c0f04ce612db0cba9a WatchSource:0}: Error finding container 66c9ee410ad682a8c546691b1bfe13ea7f49ed7546b175c0f04ce612db0cba9a: Status 404 returned error can't find the container with id 66c9ee410ad682a8c546691b1bfe13ea7f49ed7546b175c0f04ce612db0cba9a Nov 01 01:01:10.708508 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.708467 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" event={"ID":"092dea27-ada1-4838-ab3b-a38b4dbc4d97","Type":"ContainerStarted","Data":"09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800"} Nov 01 01:01:10.709019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.708539 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:10.709019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.708566 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" event={"ID":"092dea27-ada1-4838-ab3b-a38b4dbc4d97","Type":"ContainerStarted","Data":"66c9ee410ad682a8c546691b1bfe13ea7f49ed7546b175c0f04ce612db0cba9a"} Nov 01 01:01:10.743302 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:01:10.743253 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c47c543_f8cd_4a7c_811f_2f884e97a10b.slice/crio-58e82058689e90218f850783ece977c08e8e4d8d97b1cbbca294369bd878e6ac WatchSource:0}: Error finding container 58e82058689e90218f850783ece977c08e8e4d8d97b1cbbca294369bd878e6ac: Status 404 returned error can't find the container with id 58e82058689e90218f850783ece977c08e8e4d8d97b1cbbca294369bd878e6ac Nov 01 01:01:10.743458 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.743344 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf"] Nov 01 01:01:10.827836 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:10.827786 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" podStartSLOduration=1.827766641 podStartE2EDuration="1.827766641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:10.826393159 +0000 UTC m=+2215.701614175" watchObservedRunningTime="2026-11-01 01:01:10.827766641 +0000 UTC m=+2215.702987658" Nov 01 01:01:11.720114 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:11.720078 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" event={"ID":"9c47c543-f8cd-4a7c-811f-2f884e97a10b","Type":"ContainerStarted","Data":"5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86"} Nov 01 01:01:11.720663 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:11.720137 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" event={"ID":"9c47c543-f8cd-4a7c-811f-2f884e97a10b","Type":"ContainerStarted","Data":"58e82058689e90218f850783ece977c08e8e4d8d97b1cbbca294369bd878e6ac"} Nov 01 01:01:11.720663 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:11.720170 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:11.839169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:11.839117 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" podStartSLOduration=2.839104956 podStartE2EDuration="2.839104956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:11.838758948 +0000 UTC m=+2216.713979963" watchObservedRunningTime="2026-11-01 01:01:11.839104956 +0000 UTC m=+2216.714325971" Nov 01 01:01:13.936998 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:13.935158 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:19.111768 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:19.111732 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:19.389732 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:19.389645 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:19.507253 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:01:19.507214 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:01:21.194704 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:21.194664 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:21.723605 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:21.723576 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:01:22.732076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:22.732049 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:01:24.235720 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:24.235686 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:28.897771 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:28.897720 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:33.176495 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:33.176460 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:33.715979 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:33.715893 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:38.556757 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:38.556715 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:43.381016 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:43.380967 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:43.593697 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:43.593660 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:43.901220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:43.901186 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:46.562164 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:46.562125 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:47.930344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:47.925851 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:48.139651 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:48.139619 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:48.769069 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:48.769032 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:49.509327 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:01:49.509273 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:01:52.499943 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:52.499899 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:52.711503 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:52.711464 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:53.294562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.294529 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:53.450647 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.450614 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl"] Nov 01 01:01:53.450846 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.450824 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" containerID="cri-o://004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c" gracePeriod=30 Nov 01 01:01:53.560912 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.560806 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:53.563461 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.563436 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k"] Nov 01 01:01:53.563689 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.563664 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" containerID="cri-o://854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458" gracePeriod=30 Nov 01 01:01:53.568586 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.568566 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82"] Nov 01 01:01:53.568753 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.568735 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" containerID="cri-o://df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750" gracePeriod=30 Nov 01 01:01:53.932860 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:53.932767 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:55.133619 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:55.133583 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:01:55.134056 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:55.133650 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:01:55.761685 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:55.761650 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:55.973103 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:55.973064 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:56.110094 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.110059 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27"] Nov 01 01:01:56.116706 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.116667 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:56.175475 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.175411 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27"] Nov 01 01:01:56.221353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.220880 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-72acf4-dockercfg-zkvpf\"" Nov 01 01:01:56.221353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.221028 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-72acf4\"" Nov 01 01:01:56.223490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.223456 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394c5e6a-ff94-4550-b421-7686fa50097b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27\" (UID: \"394c5e6a-ff94-4550-b421-7686fa50097b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:56.223668 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.223642 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkxxj\" (UniqueName: \"kubernetes.io/projected/394c5e6a-ff94-4550-b421-7686fa50097b-kube-api-access-gkxxj\") pod \"payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27\" (UID: \"394c5e6a-ff94-4550-b421-7686fa50097b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:56.261638 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.261596 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc"] Nov 01 01:01:56.267822 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.267546 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:56.325687 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.325643 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cbh\" (UniqueName: \"kubernetes.io/projected/91185e42-540e-4fd4-bf03-f52783251c1f-kube-api-access-76cbh\") pod \"payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc\" (UID: \"91185e42-540e-4fd4-bf03-f52783251c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:56.325876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.325696 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394c5e6a-ff94-4550-b421-7686fa50097b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27\" (UID: \"394c5e6a-ff94-4550-b421-7686fa50097b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:56.325876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.325777 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91185e42-540e-4fd4-bf03-f52783251c1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc\" (UID: \"91185e42-540e-4fd4-bf03-f52783251c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:56.325876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.325831 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc"] Nov 01 01:01:56.326005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.325909 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkxxj\" (UniqueName: \"kubernetes.io/projected/394c5e6a-ff94-4550-b421-7686fa50097b-kube-api-access-gkxxj\") pod \"payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27\" (UID: \"394c5e6a-ff94-4550-b421-7686fa50097b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:56.327034 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.327005 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394c5e6a-ff94-4550-b421-7686fa50097b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27\" (UID: \"394c5e6a-ff94-4550-b421-7686fa50097b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:56.389111 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.389028 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkxxj\" (UniqueName: \"kubernetes.io/projected/394c5e6a-ff94-4550-b421-7686fa50097b-kube-api-access-gkxxj\") pod \"payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27\" (UID: \"394c5e6a-ff94-4550-b421-7686fa50097b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:56.427417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.427381 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cbh\" (UniqueName: \"kubernetes.io/projected/91185e42-540e-4fd4-bf03-f52783251c1f-kube-api-access-76cbh\") pod \"payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc\" (UID: \"91185e42-540e-4fd4-bf03-f52783251c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:56.427631 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.427454 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91185e42-540e-4fd4-bf03-f52783251c1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc\" (UID: \"91185e42-540e-4fd4-bf03-f52783251c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:56.428147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.428124 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91185e42-540e-4fd4-bf03-f52783251c1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc\" (UID: \"91185e42-540e-4fd4-bf03-f52783251c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:56.440401 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.440358 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:56.494208 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.494142 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cbh\" (UniqueName: \"kubernetes.io/projected/91185e42-540e-4fd4-bf03-f52783251c1f-kube-api-access-76cbh\") pod \"payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc\" (UID: \"91185e42-540e-4fd4-bf03-f52783251c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:56.594845 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.594813 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:56.847968 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:01:56.847936 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394c5e6a_ff94_4550_b421_7686fa50097b.slice/crio-390d2505fa3320fa970a192eee110c1de652b6d310ed231ff47baa63b388d22b WatchSource:0}: Error finding container 390d2505fa3320fa970a192eee110c1de652b6d310ed231ff47baa63b388d22b: Status 404 returned error can't find the container with id 390d2505fa3320fa970a192eee110c1de652b6d310ed231ff47baa63b388d22b Nov 01 01:01:56.847968 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.847943 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27"] Nov 01 01:01:56.985544 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:56.985518 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc"] Nov 01 01:01:56.985658 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:01:56.985515 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91185e42_540e_4fd4_bf03_f52783251c1f.slice/crio-9a14a7d3179d9c1457803bc0295833a23b90c678d44bb4c4f41535a3558b74f1 WatchSource:0}: Error finding container 9a14a7d3179d9c1457803bc0295833a23b90c678d44bb4c4f41535a3558b74f1: Status 404 returned error can't find the container with id 9a14a7d3179d9c1457803bc0295833a23b90c678d44bb4c4f41535a3558b74f1 Nov 01 01:01:57.098782 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.098740 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" event={"ID":"394c5e6a-ff94-4550-b421-7686fa50097b","Type":"ContainerStarted","Data":"9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20"} Nov 01 01:01:57.098976 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.098814 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:01:57.098976 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.098836 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" event={"ID":"394c5e6a-ff94-4550-b421-7686fa50097b","Type":"ContainerStarted","Data":"390d2505fa3320fa970a192eee110c1de652b6d310ed231ff47baa63b388d22b"} Nov 01 01:01:57.100545 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.100518 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" event={"ID":"91185e42-540e-4fd4-bf03-f52783251c1f","Type":"ContainerStarted","Data":"ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52"} Nov 01 01:01:57.100689 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.100560 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" event={"ID":"91185e42-540e-4fd4-bf03-f52783251c1f","Type":"ContainerStarted","Data":"9a14a7d3179d9c1457803bc0295833a23b90c678d44bb4c4f41535a3558b74f1"} Nov 01 01:01:57.100689 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.100611 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:01:57.223188 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.223091 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" podStartSLOduration=1.223077347 podStartE2EDuration="1.223077347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:57.220508304 +0000 UTC m=+2262.095729332" watchObservedRunningTime="2026-11-01 01:01:57.223077347 +0000 UTC m=+2262.098298359" Nov 01 01:01:57.350138 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.350069 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" podStartSLOduration=1.3500447389999999 podStartE2EDuration="1.350044739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:57.340167775 +0000 UTC m=+2262.215388791" watchObservedRunningTime="2026-11-01 01:01:57.350044739 +0000 UTC m=+2262.225265755" Nov 01 01:01:57.561981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.561944 6036 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:01:57.562156 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:57.562005 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:01:58.096485 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.096444 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:01:58.096669 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.096502 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:01:58.798814 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.798782 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 01:01:58.858051 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.858024 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 01:01:58.865612 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.865581 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-envoy\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.865782 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.865676 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-socket\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.865782 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.865714 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-podinfo\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.865782 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.865766 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-credential-socket\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.865963 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.865946 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-certs\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.866074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.866057 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-token\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-token\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.866138 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.866122 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istiod-ca-cert\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.866207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.866195 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-data\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.866269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.866227 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-kube-api-access-czwpl\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-kube-api-access-czwpl\") pod \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\" (UID: \"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f\") " Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.866629 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-socket" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.866878 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-certs" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.866893 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-data" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.866944 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-credential-socket" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.867207 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istiod-ca-cert" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.867243 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.867266 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.867315 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.867353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.867330 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.870113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.870085 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-kube-api-access-czwpl" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "kube-api-access-czwpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:01:58.870448 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.870341 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-token" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:01:58.874192 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.873642 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-envoy" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.875914 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.875888 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-podinfo" pod "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" (UID: "123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:01:58.916807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.916781 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 01:01:58.968163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968129 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-token\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-token\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.968163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968167 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istiod-ca-cert\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.968435 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968262 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-token\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-token\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.968435 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968382 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-data\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.968435 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968418 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-credential-socket\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.968594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968491 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-data\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.968594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968544 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-certs\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.968702 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968639 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-kube-api-access-45594\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-kube-api-access-45594\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.968759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968700 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istiod-ca-cert\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.968759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968699 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-credential-socket" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.968759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968730 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-envoy\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.968759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968642 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istiod-ca-cert" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:01:58.968759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968745 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-data" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.969032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968762 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-certs\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.969032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968820 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-podinfo\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.969032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968850 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-kube-api-access-m7j7k\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-kube-api-access-m7j7k\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.969032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968855 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-certs" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.969032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968880 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-credential-socket\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.969032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968905 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-data" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.969032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968911 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-envoy\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.969032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.968999 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-certs" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.969489 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.969073 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istiod-ca-cert" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:01:58.969489 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.969034 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-podinfo\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.969489 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.969206 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-socket\") pod \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\" (UID: \"b281633e-b211-4af1-b2d5-d5a700d1e9e9\") " Nov 01 01:01:58.969489 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.969245 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-socket\") pod \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\" (UID: \"d1e9e512-7cdd-4952-8752-edec0ca50c4f\") " Nov 01 01:01:58.970136 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970108 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970142 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970159 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970172 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970186 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970198 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970212 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czwpl\" (UniqueName: \"kubernetes.io/projected/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-kube-api-access-czwpl\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970229 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970241 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970255 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970269 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.970443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970299 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:58.971067 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.970541 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-socket" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.971484 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.971456 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-token" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:01:58.971890 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.971655 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-credential-socket" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.971890 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.971752 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-kube-api-access-m7j7k" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "kube-api-access-m7j7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:01:58.971890 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.971864 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-token" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:01:58.973026 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.972786 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-envoy" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.973543 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.973515 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-socket" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.973650 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.973608 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-podinfo" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:01:58.973754 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.973735 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-kube-api-access-45594" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "kube-api-access-45594". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:01:58.973910 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.973877 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-envoy" pod "b281633e-b211-4af1-b2d5-d5a700d1e9e9" (UID: "b281633e-b211-4af1-b2d5-d5a700d1e9e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:01:58.974811 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:58.974784 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-podinfo" pod "d1e9e512-7cdd-4952-8752-edec0ca50c4f" (UID: "d1e9e512-7cdd-4952-8752-edec0ca50c4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:01:59.031332 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.031229 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc"] Nov 01 01:01:59.032037 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032019 6036 cpu_manager.go:414] "containerMap: removing container" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" Nov 01 01:01:59.032089 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032039 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" Nov 01 01:01:59.032089 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032058 6036 cpu_manager.go:414] "containerMap: removing container" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" Nov 01 01:01:59.032089 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032064 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" Nov 01 01:01:59.032089 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032079 6036 cpu_manager.go:414] "containerMap: removing container" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" Nov 01 01:01:59.032089 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032084 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" Nov 01 01:01:59.032238 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032192 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerName="istio-proxy" Nov 01 01:01:59.032238 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032203 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerName="istio-proxy" Nov 01 01:01:59.032238 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.032211 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerName="istio-proxy" Nov 01 01:01:59.037179 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.037160 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.071603 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.071566 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.071785 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.071720 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spb4z\" (UniqueName: \"kubernetes.io/projected/e00b5366-0556-4856-9441-ad512c669362-kube-api-access-spb4z\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.071854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.071780 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00b5366-0556-4856-9441-ad512c669362-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.072014 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.071997 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.072177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072162 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072184 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072200 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072227 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072241 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072257 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45594\" (UniqueName: \"kubernetes.io/projected/d1e9e512-7cdd-4952-8752-edec0ca50c4f-kube-api-access-45594\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072274 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072324 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072340 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7j7k\" (UniqueName: \"kubernetes.io/projected/b281633e-b211-4af1-b2d5-d5a700d1e9e9-kube-api-access-m7j7k\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072353 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e9e512-7cdd-4952-8752-edec0ca50c4f-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.072719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.072366 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281633e-b211-4af1-b2d5-d5a700d1e9e9-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:59.091134 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.091103 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-de2a95-serving-cert\"" Nov 01 01:01:59.093935 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.093902 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc"] Nov 01 01:01:59.127945 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.127915 6036 generic.go:356] "Generic (PLEG): container finished" podID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" containerID="854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458" exitCode=0 Nov 01 01:01:59.128148 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.127969 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" Nov 01 01:01:59.128148 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.127991 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" event={"ID":"d1e9e512-7cdd-4952-8752-edec0ca50c4f","Type":"ContainerDied","Data":"854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458"} Nov 01 01:01:59.128148 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.128047 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k" event={"ID":"d1e9e512-7cdd-4952-8752-edec0ca50c4f","Type":"ContainerDied","Data":"5404ed61bf842a9cc202d1619370b8899aaec4b36738ff39b3c35d16bf021325"} Nov 01 01:01:59.128148 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.128078 6036 scope.go:122] "RemoveContainer" containerID="854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458" Nov 01 01:01:59.129840 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.129817 6036 generic.go:356] "Generic (PLEG): container finished" podID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" containerID="df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750" exitCode=0 Nov 01 01:01:59.129974 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.129871 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" Nov 01 01:01:59.129974 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.129955 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" event={"ID":"b281633e-b211-4af1-b2d5-d5a700d1e9e9","Type":"ContainerDied","Data":"df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750"} Nov 01 01:01:59.130080 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.130012 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82" event={"ID":"b281633e-b211-4af1-b2d5-d5a700d1e9e9","Type":"ContainerDied","Data":"508e1af718e094095244c889cfa6cd18c63da54c1e2719caf92d9e2f3a40c7c1"} Nov 01 01:01:59.134320 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.134259 6036 generic.go:356] "Generic (PLEG): container finished" podID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" containerID="004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c" exitCode=0 Nov 01 01:01:59.134451 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.134329 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" Nov 01 01:01:59.134451 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.134430 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" event={"ID":"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f","Type":"ContainerDied","Data":"004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c"} Nov 01 01:01:59.134570 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.134471 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl" event={"ID":"123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f","Type":"ContainerDied","Data":"b88d160eb1f44b58727538a6d18b371f28cc5d6355256561b87398d4fb5a5248"} Nov 01 01:01:59.145337 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.145265 6036 scope.go:122] "RemoveContainer" containerID="854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458" Nov 01 01:01:59.146912 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:01:59.146378 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458\": container with ID starting with 854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458 not found: ID does not exist" containerID="854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458" Nov 01 01:01:59.146912 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.146421 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458"} err="failed to get container status \"854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458\": rpc error: code = NotFound desc = could not find container \"854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458\": container with ID starting with 854cd8e47ea0ed5343c5b708c4e2cd7c76a12cd9c7da74e4ae65c6240ae18458 not found: ID does not exist" Nov 01 01:01:59.146912 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.146445 6036 scope.go:122] "RemoveContainer" containerID="df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750" Nov 01 01:01:59.159848 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.159826 6036 scope.go:122] "RemoveContainer" containerID="df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750" Nov 01 01:01:59.160256 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:01:59.160222 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750\": container with ID starting with df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750 not found: ID does not exist" containerID="df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750" Nov 01 01:01:59.160488 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.160296 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750"} err="failed to get container status \"df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750\": rpc error: code = NotFound desc = could not find container \"df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750\": container with ID starting with df8fedaec79189e2f5eb55b4e5c4b05a35a40d27eddeccdf6740dae2423cd750 not found: ID does not exist" Nov 01 01:01:59.160488 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.160323 6036 scope.go:122] "RemoveContainer" containerID="004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c" Nov 01 01:01:59.173610 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.173176 6036 scope.go:122] "RemoveContainer" containerID="004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c" Nov 01 01:01:59.173759 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:01:59.173627 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c\": container with ID starting with 004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c not found: ID does not exist" containerID="004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c" Nov 01 01:01:59.173759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.173668 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c"} err="failed to get container status \"004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c\": rpc error: code = NotFound desc = could not find container \"004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c\": container with ID starting with 004855c806f3612262ee37bf64e22e1d58abfc4bc82dc7f614a76c52a8f8b32c not found: ID does not exist" Nov 01 01:01:59.174130 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.174108 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.174205 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.174188 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spb4z\" (UniqueName: \"kubernetes.io/projected/e00b5366-0556-4856-9441-ad512c669362-kube-api-access-spb4z\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.174253 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.174219 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00b5366-0556-4856-9441-ad512c669362-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.174396 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:01:59.174273 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-serving-cert: secret "maas-api-e2e-shared-b-w6-de2a95-serving-cert" not found Nov 01 01:01:59.174396 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:01:59.174391 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls podName:e00b5366-0556-4856-9441-ad512c669362 nodeName:}" failed. No retries permitted until 2026-11-01 01:01:59.67436723 +0000 UTC m=+2264.549588247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls") pod "maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" (UID: "e00b5366-0556-4856-9441-ad512c669362") : secret "maas-api-e2e-shared-b-w6-de2a95-serving-cert" not found Nov 01 01:01:59.174941 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.174916 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00b5366-0556-4856-9441-ad512c669362-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.175182 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.175162 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.178069 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.178034 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.239119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.239088 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spb4z\" (UniqueName: \"kubernetes.io/projected/e00b5366-0556-4856-9441-ad512c669362-kube-api-access-spb4z\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.330946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.330909 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:59.511793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.511748 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82"] Nov 01 01:01:59.516240 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.516209 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-zrp82"] Nov 01 01:01:59.550007 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.549977 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:01:59.628619 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.628541 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k"] Nov 01 01:01:59.632477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.632454 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-d849k"] Nov 01 01:01:59.682936 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.682907 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.685555 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.685522 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:01:59.744854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.744820 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl"] Nov 01 01:01:59.748466 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.748436 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-tpvrl"] Nov 01 01:01:59.951928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:01:59.951843 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:02:00.348739 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:00.342659 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc"] Nov 01 01:02:00.350907 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:02:00.350871 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00b5366_0556_4856_9441_ad512c669362.slice/crio-bc6fdfb6542321db1306c7ed3b4421cf05f95887ee9250c2a05841867cfe140d WatchSource:0}: Error finding container bc6fdfb6542321db1306c7ed3b4421cf05f95887ee9250c2a05841867cfe140d: Status 404 returned error can't find the container with id bc6fdfb6542321db1306c7ed3b4421cf05f95887ee9250c2a05841867cfe140d Nov 01 01:02:01.154519 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:01.154483 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" event={"ID":"e00b5366-0556-4856-9441-ad512c669362","Type":"ContainerStarted","Data":"bc6fdfb6542321db1306c7ed3b4421cf05f95887ee9250c2a05841867cfe140d"} Nov 01 01:02:01.257479 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:01.257440 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f" path="/var/lib/kubelet/pods/123f4e51-b22a-4a7b-bc92-f3fefd1b2f3f/volumes" Nov 01 01:02:01.258210 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:01.258186 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b281633e-b211-4af1-b2d5-d5a700d1e9e9" path="/var/lib/kubelet/pods/b281633e-b211-4af1-b2d5-d5a700d1e9e9/volumes" Nov 01 01:02:01.258889 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:01.258862 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e9e512-7cdd-4952-8752-edec0ca50c4f" path="/var/lib/kubelet/pods/d1e9e512-7cdd-4952-8752-edec0ca50c4f/volumes" Nov 01 01:02:02.164725 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:02.164684 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" event={"ID":"e00b5366-0556-4856-9441-ad512c669362","Type":"ContainerStarted","Data":"1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0"} Nov 01 01:02:02.165120 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:02.164781 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:02:02.287891 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:02.287837 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" podStartSLOduration=3.408592376 podStartE2EDuration="4.287820224s" totalImagesPullingTime="879.227848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:02:02.284934143 +0000 UTC m=+2267.160155159" watchObservedRunningTime="2026-11-01 01:02:02.287820224 +0000 UTC m=+2267.163041239" Nov 01 01:02:04.640933 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:04.640887 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:08.061439 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:08.061399 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:08.112853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:08.112823 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:02:08.113019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:08.112901 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:02:08.177421 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:08.177391 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:02:08.300970 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:08.300934 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:09.960060 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:09.960013 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:12.299367 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:12.299335 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9"] Nov 01 01:02:12.299850 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:12.299582 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" containerID="cri-o://458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3" gracePeriod=30 Nov 01 01:02:12.350421 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:12.350387 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk"] Nov 01 01:02:12.350647 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:12.350599 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" containerID="cri-o://a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91" gracePeriod=30 Nov 01 01:02:14.385427 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:14.385390 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:02:14.385867 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:14.385481 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:02:14.795671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:14.795631 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:15.019664 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:15.019562 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:15.235584 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:15.235477 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:15.677644 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:15.677612 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:02:15.678049 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:15.677673 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:02:17.684984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.684957 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 01:02:17.705355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.705331 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 01:02:17.779831 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.779787 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-kube-api-access-fcvx7\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-kube-api-access-fcvx7\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.780045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.779837 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istiod-ca-cert\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.780045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.779865 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9027416f-3c14-453c-ae29-81c0c7819e2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9027416f-3c14-453c-ae29-81c0c7819e2d-istiod-ca-cert\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.780045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.779904 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-token\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-token\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.780045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.779938 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-data\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.780045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.779969 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-data\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.780045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.779999 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-kube-api-access-9wrb5\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-kube-api-access-9wrb5\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.780450 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780271 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istiod-ca-cert" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:17.780507 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780396 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-data" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.780507 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780406 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-data" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.780624 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780136 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-envoy\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.780624 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780596 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-podinfo\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.780729 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780719 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-certs\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.780856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780826 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-podinfo\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.780927 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780891 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-envoy\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.780927 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780919 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-socket\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.781037 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.780951 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-certs\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.781097 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781028 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-certs" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.781097 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781047 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-credential-socket\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.781201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781168 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-socket\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.781261 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781234 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-socket" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.781261 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781243 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-credential-socket\") pod \"9027416f-3c14-453c-ae29-81c0c7819e2d\" (UID: \"9027416f-3c14-453c-ae29-81c0c7819e2d\") " Nov 01 01:02:17.781261 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781249 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-certs" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.781467 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781307 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-token\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-token\") pod \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\" (UID: \"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc\") " Nov 01 01:02:17.781522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781469 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-credential-socket" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.781522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781486 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-socket" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781692 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-credential-socket" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.781922 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9027416f-3c14-453c-ae29-81c0c7819e2d-istiod-ca-cert" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782309 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782328 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782342 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782356 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782368 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782380 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782392 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9027416f-3c14-453c-ae29-81c0c7819e2d-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782404 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782416 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.782776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.782428 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.783459 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.783355 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-kube-api-access-fcvx7" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "kube-api-access-fcvx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:17.785086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.784908 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-podinfo" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:02:17.785193 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.784987 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-podinfo" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:02:17.785193 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.785113 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-kube-api-access-9wrb5" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "kube-api-access-9wrb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:17.785340 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.785240 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-token" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:17.785509 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.785476 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-envoy" pod "9027416f-3c14-453c-ae29-81c0c7819e2d" (UID: "9027416f-3c14-453c-ae29-81c0c7819e2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.785909 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.785883 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-token" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:17.786647 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.786627 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-envoy" pod "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" (UID: "bf10d2f9-c3ea-43e1-bb2e-9345697fbccc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:17.884199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.884103 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.884199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.884136 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.884199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.884150 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.884199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.884162 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcvx7\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-kube-api-access-fcvx7\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.884199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.884176 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.884199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.884188 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrb5\" (UniqueName: \"kubernetes.io/projected/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-kube-api-access-9wrb5\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.884199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.884201 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9027416f-3c14-453c-ae29-81c0c7819e2d-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.884581 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:17.884214 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:18.310640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.310607 6036 generic.go:356] "Generic (PLEG): container finished" podID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerID="a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91" exitCode=0 Nov 01 01:02:18.310825 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.310661 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" Nov 01 01:02:18.310825 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.310697 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" event={"ID":"9027416f-3c14-453c-ae29-81c0c7819e2d","Type":"ContainerDied","Data":"a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91"} Nov 01 01:02:18.310825 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.310759 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk" event={"ID":"9027416f-3c14-453c-ae29-81c0c7819e2d","Type":"ContainerDied","Data":"3ec9f7605f02739daace9d39b0819d3c84ff84a2e6e65276f3a84f6df5061521"} Nov 01 01:02:18.310825 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.310791 6036 scope.go:122] "RemoveContainer" containerID="a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91" Nov 01 01:02:18.312240 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.312217 6036 generic.go:356] "Generic (PLEG): container finished" podID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerID="458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3" exitCode=0 Nov 01 01:02:18.312376 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.312272 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" Nov 01 01:02:18.312376 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.312313 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" event={"ID":"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc","Type":"ContainerDied","Data":"458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3"} Nov 01 01:02:18.312376 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.312360 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9" event={"ID":"bf10d2f9-c3ea-43e1-bb2e-9345697fbccc","Type":"ContainerDied","Data":"bebf3f777eb78d1ce2f4e8740c550db06decaff65b312736c04360654460985d"} Nov 01 01:02:18.339950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.339931 6036 scope.go:122] "RemoveContainer" containerID="a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91" Nov 01 01:02:18.340266 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:02:18.340248 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91\": container with ID starting with a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91 not found: ID does not exist" containerID="a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91" Nov 01 01:02:18.340351 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.340279 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91"} err="failed to get container status \"a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91\": rpc error: code = NotFound desc = could not find container \"a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91\": container with ID starting with a284fb48af1b1dae0e705eabb68b1830672a813b389e79c37466e0c68a9edd91 not found: ID does not exist" Nov 01 01:02:18.340351 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.340321 6036 scope.go:122] "RemoveContainer" containerID="458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3" Nov 01 01:02:18.350176 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.350159 6036 scope.go:122] "RemoveContainer" containerID="458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3" Nov 01 01:02:18.350476 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:02:18.350457 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3\": container with ID starting with 458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3 not found: ID does not exist" containerID="458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3" Nov 01 01:02:18.350550 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.350491 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3"} err="failed to get container status \"458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3\": rpc error: code = NotFound desc = could not find container \"458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3\": container with ID starting with 458a0a0697516db6e569947d33950e73d12316372b706b7bd43bde3a734e91a3 not found: ID does not exist" Nov 01 01:02:18.480721 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.480686 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk"] Nov 01 01:02:18.531505 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.531472 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-jgqwk"] Nov 01 01:02:18.597206 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.597113 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9"] Nov 01 01:02:18.600333 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:18.600305 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-5w8p9"] Nov 01 01:02:19.251980 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:19.251941 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" path="/var/lib/kubelet/pods/9027416f-3c14-453c-ae29-81c0c7819e2d/volumes" Nov 01 01:02:19.252513 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:19.252497 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" path="/var/lib/kubelet/pods/bf10d2f9-c3ea-43e1-bb2e-9345697fbccc/volumes" Nov 01 01:02:19.505544 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:02:19.505459 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:02:20.821201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:20.821152 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:24.665501 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:24.665468 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9"] Nov 01 01:02:25.084199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:25.084130 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:25.295700 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:25.295653 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:26.375240 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:26.375208 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:29.666660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:29.666613 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" podUID="82a400a2-938a-4ff3-9c1e-5c0f51baf810" containerName="payload-pre-processing" containerID="cri-o://3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1" gracePeriod=25 Nov 01 01:02:29.965665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:29.965633 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 01:02:29.992841 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:29.992808 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a400a2-938a-4ff3-9c1e-5c0f51baf810-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a400a2-938a-4ff3-9c1e-5c0f51baf810-plugins-config-volume\") pod \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\" (UID: \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\") " Nov 01 01:02:29.993167 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:29.993149 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a400a2-938a-4ff3-9c1e-5c0f51baf810-kube-api-access-plh4b\" (UniqueName: \"kubernetes.io/projected/82a400a2-938a-4ff3-9c1e-5c0f51baf810-kube-api-access-plh4b\") pod \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\" (UID: \"82a400a2-938a-4ff3-9c1e-5c0f51baf810\") " Nov 01 01:02:29.993261 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:29.993237 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a400a2-938a-4ff3-9c1e-5c0f51baf810-plugins-config-volume" pod "82a400a2-938a-4ff3-9c1e-5c0f51baf810" (UID: "82a400a2-938a-4ff3-9c1e-5c0f51baf810"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:29.993670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:29.993655 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a400a2-938a-4ff3-9c1e-5c0f51baf810-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:29.995599 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:29.995574 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a400a2-938a-4ff3-9c1e-5c0f51baf810-kube-api-access-plh4b" pod "82a400a2-938a-4ff3-9c1e-5c0f51baf810" (UID: "82a400a2-938a-4ff3-9c1e-5c0f51baf810"). InnerVolumeSpecName "kube-api-access-plh4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:30.095348 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.095271 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plh4b\" (UniqueName: \"kubernetes.io/projected/82a400a2-938a-4ff3-9c1e-5c0f51baf810-kube-api-access-plh4b\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:30.412860 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.412822 6036 generic.go:356] "Generic (PLEG): container finished" podID="82a400a2-938a-4ff3-9c1e-5c0f51baf810" containerID="3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1" exitCode=0 Nov 01 01:02:30.413084 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.412884 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" Nov 01 01:02:30.413084 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.412892 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" event={"ID":"82a400a2-938a-4ff3-9c1e-5c0f51baf810","Type":"ContainerDied","Data":"3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1"} Nov 01 01:02:30.413084 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.412970 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9" event={"ID":"82a400a2-938a-4ff3-9c1e-5c0f51baf810","Type":"ContainerDied","Data":"28555382b9d5ab8cd968d98556695dc8ddcf4f191b8822810425d22f84de092a"} Nov 01 01:02:30.413084 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.412992 6036 scope.go:122] "RemoveContainer" containerID="3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1" Nov 01 01:02:30.424751 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.424729 6036 scope.go:122] "RemoveContainer" containerID="3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1" Nov 01 01:02:30.425036 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:02:30.425011 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1\": container with ID starting with 3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1 not found: ID does not exist" containerID="3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1" Nov 01 01:02:30.425115 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.425044 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1"} err="failed to get container status \"3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1\": rpc error: code = NotFound desc = could not find container \"3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1\": container with ID starting with 3704c51205ab1d5115afc515896dfd50bd8e78f8de05bd12aa65ab6e0b6da0e1 not found: ID does not exist" Nov 01 01:02:30.471071 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:02:30.471032 6036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a400a2_938a_4ff3_9c1e_5c0f51baf810.slice/crio-28555382b9d5ab8cd968d98556695dc8ddcf4f191b8822810425d22f84de092a\": RecentStats: unable to find data in memory cache]" Nov 01 01:02:30.584526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.584487 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9"] Nov 01 01:02:30.635100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:30.635066 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af82f1-7478f447f9-72pl9"] Nov 01 01:02:31.253220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:31.253187 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a400a2-938a-4ff3-9c1e-5c0f51baf810" path="/var/lib/kubelet/pods/82a400a2-938a-4ff3-9c1e-5c0f51baf810/volumes" Nov 01 01:02:32.106813 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:32.106727 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:33.532691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:33.532654 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:37.145745 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:37.145713 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:42.163545 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:42.163466 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:42.374115 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:42.374077 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:42.589649 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:42.589608 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:47.463123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:47.463088 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:49.506136 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:02:49.506099 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:02:52.266050 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:52.266004 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:53.929616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:53.929578 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj"] Nov 01 01:02:53.930072 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:53.929781 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" podUID="b0c228fd-2e0f-4c82-a177-1bae31193c6a" containerName="discovery" containerID="cri-o://8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5" gracePeriod=30 Nov 01 01:02:54.222242 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.222212 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 01:02:54.364613 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.364581 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-ca-configmap\") pod \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " Nov 01 01:02:54.364802 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.364673 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-token\") pod \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " Nov 01 01:02:54.364802 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.364790 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-cacerts\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-cacerts\") pod \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " Nov 01 01:02:54.364937 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.364870 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-kube-api-access-vf7dp\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-kube-api-access-vf7dp\") pod \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " Nov 01 01:02:54.364998 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.364978 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c228fd-2e0f-4c82-a177-1bae31193c6a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c228fd-2e0f-4c82-a177-1bae31193c6a-local-certs\") pod \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " Nov 01 01:02:54.365124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.365066 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-ca-configmap" pod "b0c228fd-2e0f-4c82-a177-1bae31193c6a" (UID: "b0c228fd-2e0f-4c82-a177-1bae31193c6a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:54.365124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.365116 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-dns-cert\") pod \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " Nov 01 01:02:54.365389 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.365161 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-kubeconfig\") pod \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\" (UID: \"b0c228fd-2e0f-4c82-a177-1bae31193c6a\") " Nov 01 01:02:54.365915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.365894 6036 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-ca-configmap\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:54.367810 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.367782 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c228fd-2e0f-4c82-a177-1bae31193c6a-local-certs" pod "b0c228fd-2e0f-4c82-a177-1bae31193c6a" (UID: "b0c228fd-2e0f-4c82-a177-1bae31193c6a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:54.368220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.368194 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-dns-cert" pod "b0c228fd-2e0f-4c82-a177-1bae31193c6a" (UID: "b0c228fd-2e0f-4c82-a177-1bae31193c6a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:02:54.368559 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.368519 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-kube-api-access-vf7dp" pod "b0c228fd-2e0f-4c82-a177-1bae31193c6a" (UID: "b0c228fd-2e0f-4c82-a177-1bae31193c6a"). InnerVolumeSpecName "kube-api-access-vf7dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:54.369823 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.369762 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-token" pod "b0c228fd-2e0f-4c82-a177-1bae31193c6a" (UID: "b0c228fd-2e0f-4c82-a177-1bae31193c6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:54.369823 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.369777 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-cacerts" pod "b0c228fd-2e0f-4c82-a177-1bae31193c6a" (UID: "b0c228fd-2e0f-4c82-a177-1bae31193c6a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:02:54.370220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.370198 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-kubeconfig" pod "b0c228fd-2e0f-4c82-a177-1bae31193c6a" (UID: "b0c228fd-2e0f-4c82-a177-1bae31193c6a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:02:54.467691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.467607 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:54.467691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.467637 6036 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-cacerts\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:54.467691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.467647 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf7dp\" (UniqueName: \"kubernetes.io/projected/b0c228fd-2e0f-4c82-a177-1bae31193c6a-kube-api-access-vf7dp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:54.467691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.467657 6036 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c228fd-2e0f-4c82-a177-1bae31193c6a-local-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:54.467691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.467668 6036 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-csr-dns-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:54.467691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.467676 6036 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0c228fd-2e0f-4c82-a177-1bae31193c6a-istio-kubeconfig\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:54.601740 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.601702 6036 generic.go:356] "Generic (PLEG): container finished" podID="b0c228fd-2e0f-4c82-a177-1bae31193c6a" containerID="8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5" exitCode=0 Nov 01 01:02:54.601911 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.601760 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" Nov 01 01:02:54.601911 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.601789 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" event={"ID":"b0c228fd-2e0f-4c82-a177-1bae31193c6a","Type":"ContainerDied","Data":"8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5"} Nov 01 01:02:54.601911 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.601863 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj" event={"ID":"b0c228fd-2e0f-4c82-a177-1bae31193c6a","Type":"ContainerDied","Data":"d750ba4d51a3eb007bd0c88feba2b5fa5f2cb7cf079463c91490fcfbf282c78d"} Nov 01 01:02:54.601911 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.601889 6036 scope.go:122] "RemoveContainer" containerID="8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5" Nov 01 01:02:54.613992 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.613972 6036 scope.go:122] "RemoveContainer" containerID="8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5" Nov 01 01:02:54.614315 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:02:54.614296 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5\": container with ID starting with 8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5 not found: ID does not exist" containerID="8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5" Nov 01 01:02:54.614387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.614322 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5"} err="failed to get container status \"8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5\": rpc error: code = NotFound desc = could not find container \"8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5\": container with ID starting with 8245d5afc2460f7699c735f61ec1db9e44ea588855ec53f889a6e7ac3386b5e5 not found: ID does not exist" Nov 01 01:02:54.771909 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.771812 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj"] Nov 01 01:02:54.822421 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:54.822387 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k98tj"] Nov 01 01:02:55.252387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:55.252356 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c228fd-2e0f-4c82-a177-1bae31193c6a" path="/var/lib/kubelet/pods/b0c228fd-2e0f-4c82-a177-1bae31193c6a/volumes" Nov 01 01:02:56.504543 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:56.504505 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:56.750846 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:56.750815 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:02:57.159503 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:02:57.159470 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:01.955748 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:01.955715 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:06.748071 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:06.748030 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:10.793690 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:10.793648 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6"] Nov 01 01:03:10.794179 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:10.793910 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" podUID="db4381d5-90f1-407d-9130-7ce9910ce004" containerName="maas-api" containerID="cri-o://7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e" gracePeriod=30 Nov 01 01:03:10.961444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:10.961407 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:11.101200 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.101173 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 01:03:11.228368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.220702 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4381d5-90f1-407d-9130-7ce9910ce004-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4381d5-90f1-407d-9130-7ce9910ce004-openshift-service-ca\") pod \"db4381d5-90f1-407d-9130-7ce9910ce004\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " Nov 01 01:03:11.228368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.220822 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-metrics-certs\") pod \"db4381d5-90f1-407d-9130-7ce9910ce004\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " Nov 01 01:03:11.228368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.220950 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4381d5-90f1-407d-9130-7ce9910ce004-kube-api-access-dl2t4\" (UniqueName: \"kubernetes.io/projected/db4381d5-90f1-407d-9130-7ce9910ce004-kube-api-access-dl2t4\") pod \"db4381d5-90f1-407d-9130-7ce9910ce004\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " Nov 01 01:03:11.228368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.221026 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls\") pod \"db4381d5-90f1-407d-9130-7ce9910ce004\" (UID: \"db4381d5-90f1-407d-9130-7ce9910ce004\") " Nov 01 01:03:11.229482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.229142 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4381d5-90f1-407d-9130-7ce9910ce004-openshift-service-ca" pod "db4381d5-90f1-407d-9130-7ce9910ce004" (UID: "db4381d5-90f1-407d-9130-7ce9910ce004"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:11.229482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.229421 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls" pod "db4381d5-90f1-407d-9130-7ce9910ce004" (UID: "db4381d5-90f1-407d-9130-7ce9910ce004"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:03:11.248085 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.248030 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4381d5-90f1-407d-9130-7ce9910ce004-kube-api-access-dl2t4" pod "db4381d5-90f1-407d-9130-7ce9910ce004" (UID: "db4381d5-90f1-407d-9130-7ce9910ce004"). InnerVolumeSpecName "kube-api-access-dl2t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:11.248400 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.248370 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-metrics-certs" pod "db4381d5-90f1-407d-9130-7ce9910ce004" (UID: "db4381d5-90f1-407d-9130-7ce9910ce004"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:03:11.323467 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.323370 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:11.323467 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.323408 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2t4\" (UniqueName: \"kubernetes.io/projected/db4381d5-90f1-407d-9130-7ce9910ce004-kube-api-access-dl2t4\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:11.323467 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.323421 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4381d5-90f1-407d-9130-7ce9910ce004-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:11.323467 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.323433 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4381d5-90f1-407d-9130-7ce9910ce004-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:11.723777 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.723682 6036 generic.go:356] "Generic (PLEG): container finished" podID="db4381d5-90f1-407d-9130-7ce9910ce004" containerID="7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e" exitCode=0 Nov 01 01:03:11.723777 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.723725 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" Nov 01 01:03:11.723777 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.723761 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" event={"ID":"db4381d5-90f1-407d-9130-7ce9910ce004","Type":"ContainerDied","Data":"7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e"} Nov 01 01:03:11.724021 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.723817 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6" event={"ID":"db4381d5-90f1-407d-9130-7ce9910ce004","Type":"ContainerDied","Data":"cbc44b9292f02580481423f71e33ec0e13787bc75e511debea7ed286b70b8bca"} Nov 01 01:03:11.724021 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.723843 6036 scope.go:122] "RemoveContainer" containerID="7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e" Nov 01 01:03:11.734831 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.734811 6036 scope.go:122] "RemoveContainer" containerID="7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e" Nov 01 01:03:11.735104 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:11.735081 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e\": container with ID starting with 7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e not found: ID does not exist" containerID="7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e" Nov 01 01:03:11.735153 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.735113 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e"} err="failed to get container status \"7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e\": rpc error: code = NotFound desc = could not find container \"7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e\": container with ID starting with 7e228dc6c26f1681e4198f5cf6e51e19c475e684fe55083266771f17c7bc7e4e not found: ID does not exist" Nov 01 01:03:11.895239 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.895201 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6"] Nov 01 01:03:11.899883 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:11.899859 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-27dee6-7c57b8dff8-kgrz6"] Nov 01 01:03:12.554302 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:12.554249 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:12.775187 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:12.775150 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:12.988108 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:12.988028 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:13.252612 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:13.252522 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4381d5-90f1-407d-9130-7ce9910ce004" path="/var/lib/kubelet/pods/db4381d5-90f1-407d-9130-7ce9910ce004/volumes" Nov 01 01:03:13.469092 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:13.469054 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:18.192775 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:18.192729 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:19.506701 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:19.506664 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:03:20.035546 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:20.035514 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:20.236529 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:20.236482 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:20.353839 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:20.353757 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn"] Nov 01 01:03:20.353997 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:20.353962 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" containerID="cri-o://5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f" gracePeriod=30 Nov 01 01:03:20.456126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:20.456005 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:22.857967 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:22.857928 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:24.020756 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:24.020675 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb"] Nov 01 01:03:24.130854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:24.130820 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54"] Nov 01 01:03:24.177299 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:24.177248 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7"] Nov 01 01:03:24.177555 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:24.177518 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" containerID="cri-o://85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3" gracePeriod=30 Nov 01 01:03:24.469558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:24.469470 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:24.677535 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:24.677485 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:25.006670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.006638 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:25.213214 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.213184 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:25.648944 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.648904 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:25.753104 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.753072 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867304 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-token\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-token\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867363 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istiod-ca-cert\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867457 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-kube-api-access-cpq96\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-kube-api-access-cpq96\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867528 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-socket\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867586 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-podinfo\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867674 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-credential-socket\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867750 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-data\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867777 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-envoy\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.867845 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-certs\") pod \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\" (UID: \"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a\") " Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.868046 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istiod-ca-cert" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:25.869390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.868542 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.872539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.872507 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-socket" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:25.872687 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.872639 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-token" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:25.887388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.884426 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-certs" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:25.887388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.884680 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-credential-socket" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:25.887388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.886319 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-data" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:25.889035 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.888998 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-kube-api-access-cpq96" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "kube-api-access-cpq96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:25.891746 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.891719 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-podinfo" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:03:25.899716 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.899683 6036 generic.go:356] "Generic (PLEG): container finished" podID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerID="5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f" exitCode=0 Nov 01 01:03:25.899855 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.899769 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" Nov 01 01:03:25.899855 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.899788 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" event={"ID":"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a","Type":"ContainerDied","Data":"5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f"} Nov 01 01:03:25.899974 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.899886 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn" event={"ID":"c9c93b96-6c9c-4cbc-901a-dc98de25ec1a","Type":"ContainerDied","Data":"7a2a062723b193fddc58ea4e7030f7238c4f0e232d8c3e6414a466f8e13a490d"} Nov 01 01:03:25.899974 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.899923 6036 scope.go:122] "RemoveContainer" containerID="5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f" Nov 01 01:03:25.906615 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.906585 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-envoy" pod "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" (UID: "c9c93b96-6c9c-4cbc-901a-dc98de25ec1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:25.917555 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.917526 6036 scope.go:122] "RemoveContainer" containerID="dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8" Nov 01 01:03:25.930405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.930382 6036 scope.go:122] "RemoveContainer" containerID="5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f" Nov 01 01:03:25.930710 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:25.930690 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f\": container with ID starting with 5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f not found: ID does not exist" containerID="5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f" Nov 01 01:03:25.930792 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.930726 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f"} err="failed to get container status \"5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f\": rpc error: code = NotFound desc = could not find container \"5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f\": container with ID starting with 5c4c5cf5894813da88750aa00a611e8e95af719fdf3d84735e9199e9e3a1bd1f not found: ID does not exist" Nov 01 01:03:25.930792 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.930755 6036 scope.go:122] "RemoveContainer" containerID="dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8" Nov 01 01:03:25.931036 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:25.931019 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8\": container with ID starting with dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8 not found: ID does not exist" containerID="dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8" Nov 01 01:03:25.931102 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.931045 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8"} err="failed to get container status \"dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8\": rpc error: code = NotFound desc = could not find container \"dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8\": container with ID starting with dba6e01ba9b514f1e6afb344a6153775d6b73c1e954e8eadd49f813df04deaf8 not found: ID does not exist" Nov 01 01:03:25.970909 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.970875 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.970909 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.970906 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.971163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.970918 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.971163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.970931 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.971163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.970942 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.971163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.970954 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpq96\" (UniqueName: \"kubernetes.io/projected/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-kube-api-access-cpq96\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.971163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.970967 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.971163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:25.970979 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:26.331490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:26.331453 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn"] Nov 01 01:03:26.381040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:26.381009 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-27dee6-openshift-default-5dccb4b79-8lqfn"] Nov 01 01:03:26.830694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:26.830663 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:03:26.830893 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:26.830734 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:03:27.252920 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:27.252836 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" path="/var/lib/kubelet/pods/c9c93b96-6c9c-4cbc-901a-dc98de25ec1a/volumes" Nov 01 01:03:29.021224 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.021143 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" podUID="8ee7203e-1654-44fc-9f9a-5c3d5d7db131" containerName="payload-processing" containerID="cri-o://86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba" gracePeriod=25 Nov 01 01:03:29.131951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.131911 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" podUID="599bb32b-a4fd-4cb6-9787-4ae7885dc770" containerName="payload-pre-processing" containerID="cri-o://cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640" gracePeriod=25 Nov 01 01:03:29.318315 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.315625 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:03:29.429877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.429842 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-plugins-config-volume\") pod \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\" (UID: \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\") " Nov 01 01:03:29.431421 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.430668 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-plugins-config-volume" pod "8ee7203e-1654-44fc-9f9a-5c3d5d7db131" (UID: "8ee7203e-1654-44fc-9f9a-5c3d5d7db131"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:29.431421 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.430735 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-kube-api-access-cqdjx\" (UniqueName: \"kubernetes.io/projected/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-kube-api-access-cqdjx\") pod \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\" (UID: \"8ee7203e-1654-44fc-9f9a-5c3d5d7db131\") " Nov 01 01:03:29.432476 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.431594 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.440419 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.440376 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-kube-api-access-cqdjx" pod "8ee7203e-1654-44fc-9f9a-5c3d5d7db131" (UID: "8ee7203e-1654-44fc-9f9a-5c3d5d7db131"). InnerVolumeSpecName "kube-api-access-cqdjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:29.484480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.484433 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:03:29.500257 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.500231 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 01:03:29.533450 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.533369 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqdjx\" (UniqueName: \"kubernetes.io/projected/8ee7203e-1654-44fc-9f9a-5c3d5d7db131-kube-api-access-cqdjx\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.634779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.634743 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599bb32b-a4fd-4cb6-9787-4ae7885dc770-kube-api-access-5qf6r\" (UniqueName: \"kubernetes.io/projected/599bb32b-a4fd-4cb6-9787-4ae7885dc770-kube-api-access-5qf6r\") pod \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\" (UID: \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\") " Nov 01 01:03:29.634939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.634870 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599bb32b-a4fd-4cb6-9787-4ae7885dc770-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599bb32b-a4fd-4cb6-9787-4ae7885dc770-plugins-config-volume\") pod \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\" (UID: \"599bb32b-a4fd-4cb6-9787-4ae7885dc770\") " Nov 01 01:03:29.634939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.634917 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-istio-token\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-istio-token\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.635099 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.634986 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-data\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.635099 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.635046 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e00ff82-0096-4553-9f58-43b412f61f30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e00ff82-0096-4553-9f58-43b412f61f30-istio-podinfo\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.635349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.635325 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599bb32b-a4fd-4cb6-9787-4ae7885dc770-plugins-config-volume" pod "599bb32b-a4fd-4cb6-9787-4ae7885dc770" (UID: "599bb32b-a4fd-4cb6-9787-4ae7885dc770"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:29.635472 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.635436 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-data" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:29.635530 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.635499 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e00ff82-0096-4553-9f58-43b412f61f30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e00ff82-0096-4553-9f58-43b412f61f30-istiod-ca-cert\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.635737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.635717 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-certs\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.635818 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.635752 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e00ff82-0096-4553-9f58-43b412f61f30-istiod-ca-cert" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:29.636047 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.635990 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-certs" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.636133 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-socket\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.636190 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-kube-api-access-hq4m7\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-kube-api-access-hq4m7\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.636223 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-credential-socket\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.636379 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-envoy\") pod \"2e00ff82-0096-4553-9f58-43b412f61f30\" (UID: \"2e00ff82-0096-4553-9f58-43b412f61f30\") " Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.636733 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-credential-socket" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.637104 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599bb32b-a4fd-4cb6-9787-4ae7885dc770-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.637111 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-socket" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.637128 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.637144 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e00ff82-0096-4553-9f58-43b412f61f30-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.637157 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.637196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.637171 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.638094 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.638070 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e00ff82-0096-4553-9f58-43b412f61f30-istio-podinfo" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:03:29.639333 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.639276 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-istio-token" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:29.639559 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.639523 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-envoy" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:03:29.639677 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.639653 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599bb32b-a4fd-4cb6-9787-4ae7885dc770-kube-api-access-5qf6r" pod "599bb32b-a4fd-4cb6-9787-4ae7885dc770" (UID: "599bb32b-a4fd-4cb6-9787-4ae7885dc770"). InnerVolumeSpecName "kube-api-access-5qf6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:29.639935 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.639911 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-kube-api-access-hq4m7" pod "2e00ff82-0096-4553-9f58-43b412f61f30" (UID: "2e00ff82-0096-4553-9f58-43b412f61f30"). InnerVolumeSpecName "kube-api-access-hq4m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:29.739054 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.739018 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qf6r\" (UniqueName: \"kubernetes.io/projected/599bb32b-a4fd-4cb6-9787-4ae7885dc770-kube-api-access-5qf6r\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.739054 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.739055 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.739331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.739076 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e00ff82-0096-4553-9f58-43b412f61f30-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.739331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.739091 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.739331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.739107 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq4m7\" (UniqueName: \"kubernetes.io/projected/2e00ff82-0096-4553-9f58-43b412f61f30-kube-api-access-hq4m7\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.739331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.739120 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e00ff82-0096-4553-9f58-43b412f61f30-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:29.933783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.933690 6036 generic.go:356] "Generic (PLEG): container finished" podID="2e00ff82-0096-4553-9f58-43b412f61f30" containerID="85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3" exitCode=0 Nov 01 01:03:29.933783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.933739 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" Nov 01 01:03:29.933783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.933773 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" event={"ID":"2e00ff82-0096-4553-9f58-43b412f61f30","Type":"ContainerDied","Data":"85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3"} Nov 01 01:03:29.934071 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.933822 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7" event={"ID":"2e00ff82-0096-4553-9f58-43b412f61f30","Type":"ContainerDied","Data":"dd42e8157dfaec6f16f89e5bc90206bc962316701ff53e2e47256475e5e5eb01"} Nov 01 01:03:29.934071 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.933848 6036 scope.go:122] "RemoveContainer" containerID="85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3" Nov 01 01:03:29.940321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.937589 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" Nov 01 01:03:29.940321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.937546 6036 generic.go:356] "Generic (PLEG): container finished" podID="8ee7203e-1654-44fc-9f9a-5c3d5d7db131" containerID="86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba" exitCode=0 Nov 01 01:03:29.940321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.937634 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" event={"ID":"8ee7203e-1654-44fc-9f9a-5c3d5d7db131","Type":"ContainerDied","Data":"86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba"} Nov 01 01:03:29.940321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.937733 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb" event={"ID":"8ee7203e-1654-44fc-9f9a-5c3d5d7db131","Type":"ContainerDied","Data":"35e86a73add1090d001b498cb1e014be40331283e9abab36ad1073c6015f5d46"} Nov 01 01:03:29.940321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.939593 6036 generic.go:356] "Generic (PLEG): container finished" podID="599bb32b-a4fd-4cb6-9787-4ae7885dc770" containerID="cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640" exitCode=0 Nov 01 01:03:29.940321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.939624 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" event={"ID":"599bb32b-a4fd-4cb6-9787-4ae7885dc770","Type":"ContainerDied","Data":"cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640"} Nov 01 01:03:29.940321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.939649 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" Nov 01 01:03:29.940321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.939660 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54" event={"ID":"599bb32b-a4fd-4cb6-9787-4ae7885dc770","Type":"ContainerDied","Data":"e1fe682b91ab88c751c1365719f98f69b9f1f8239213a7e8fca79db0f90952cc"} Nov 01 01:03:29.956504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.956476 6036 scope.go:122] "RemoveContainer" containerID="85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3" Nov 01 01:03:29.957132 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:29.957103 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3\": container with ID starting with 85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3 not found: ID does not exist" containerID="85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3" Nov 01 01:03:29.957333 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.957275 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3"} err="failed to get container status \"85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3\": rpc error: code = NotFound desc = could not find container \"85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3\": container with ID starting with 85622889e814a0408f06f7688f5cd81fffcb7b40025f92681aa8a9c013cdb3c3 not found: ID does not exist" Nov 01 01:03:29.957523 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.957508 6036 scope.go:122] "RemoveContainer" containerID="86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba" Nov 01 01:03:29.972849 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.972826 6036 scope.go:122] "RemoveContainer" containerID="86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba" Nov 01 01:03:29.973144 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:29.973123 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba\": container with ID starting with 86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba not found: ID does not exist" containerID="86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba" Nov 01 01:03:29.973189 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.973152 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba"} err="failed to get container status \"86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba\": rpc error: code = NotFound desc = could not find container \"86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba\": container with ID starting with 86fc476866e79b0a53b90cc2f2a0568839e73531144fc521964d8871e27524ba not found: ID does not exist" Nov 01 01:03:29.973189 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.973170 6036 scope.go:122] "RemoveContainer" containerID="cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640" Nov 01 01:03:29.983002 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.982977 6036 scope.go:122] "RemoveContainer" containerID="cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640" Nov 01 01:03:29.983336 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:29.983266 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640\": container with ID starting with cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640 not found: ID does not exist" containerID="cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640" Nov 01 01:03:29.983429 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:29.983408 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640"} err="failed to get container status \"cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640\": rpc error: code = NotFound desc = could not find container \"cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640\": container with ID starting with cc1fbfafc530c92b1300fd37edc06f445836f47986dfa9b4e2a7e7788305f640 not found: ID does not exist" Nov 01 01:03:30.105271 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:30.105234 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54"] Nov 01 01:03:30.108648 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:30.108621 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f9d81-5d4b5799-5wt54"] Nov 01 01:03:30.164247 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:30.164210 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:30.219558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:30.219468 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb"] Nov 01 01:03:30.224950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:30.224920 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f9d81-6558dfbb44-k9nxb"] Nov 01 01:03:30.335933 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:30.335899 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7"] Nov 01 01:03:30.341730 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:30.341701 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-xf9m7"] Nov 01 01:03:31.253912 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:31.253877 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" path="/var/lib/kubelet/pods/2e00ff82-0096-4553-9f58-43b412f61f30/volumes" Nov 01 01:03:31.254453 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:31.254428 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599bb32b-a4fd-4cb6-9787-4ae7885dc770" path="/var/lib/kubelet/pods/599bb32b-a4fd-4cb6-9787-4ae7885dc770/volumes" Nov 01 01:03:31.254861 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:31.254844 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee7203e-1654-44fc-9f9a-5c3d5d7db131" path="/var/lib/kubelet/pods/8ee7203e-1654-44fc-9f9a-5c3d5d7db131/volumes" Nov 01 01:03:32.353065 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:32.353017 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:32.576247 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:32.576217 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:32.792477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:32.792405 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:33.007272 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:33.007237 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:33.216620 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:33.216534 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:33.880612 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:33.880572 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:35.456175 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:35.456138 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:38.183125 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:38.183087 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj"] Nov 01 01:03:38.292789 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:38.292750 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d"] Nov 01 01:03:38.604325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:38.602714 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:38.810382 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:38.810351 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:41.047961 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:41.047921 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:43.183551 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.183506 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" podUID="eac98067-e982-45a1-8b6a-022e2e331657" containerName="payload-processing" containerID="cri-o://b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915" gracePeriod=25 Nov 01 01:03:43.293696 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.293655 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" podUID="2919d823-3643-43f7-9a68-4640fdfc25c8" containerName="payload-pre-processing" containerID="cri-o://f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90" gracePeriod=25 Nov 01 01:03:43.483102 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.482461 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 01:03:43.551817 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.551785 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac98067-e982-45a1-8b6a-022e2e331657-kube-api-access-nwcqt\" (UniqueName: \"kubernetes.io/projected/eac98067-e982-45a1-8b6a-022e2e331657-kube-api-access-nwcqt\") pod \"eac98067-e982-45a1-8b6a-022e2e331657\" (UID: \"eac98067-e982-45a1-8b6a-022e2e331657\") " Nov 01 01:03:43.552022 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.551975 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac98067-e982-45a1-8b6a-022e2e331657-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac98067-e982-45a1-8b6a-022e2e331657-plugins-config-volume\") pod \"eac98067-e982-45a1-8b6a-022e2e331657\" (UID: \"eac98067-e982-45a1-8b6a-022e2e331657\") " Nov 01 01:03:43.552352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.552325 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac98067-e982-45a1-8b6a-022e2e331657-plugins-config-volume" pod "eac98067-e982-45a1-8b6a-022e2e331657" (UID: "eac98067-e982-45a1-8b6a-022e2e331657"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:43.552564 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.552550 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac98067-e982-45a1-8b6a-022e2e331657-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:43.554517 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.554495 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac98067-e982-45a1-8b6a-022e2e331657-kube-api-access-nwcqt" pod "eac98067-e982-45a1-8b6a-022e2e331657" (UID: "eac98067-e982-45a1-8b6a-022e2e331657"). InnerVolumeSpecName "kube-api-access-nwcqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:43.575584 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.575557 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 01:03:43.653322 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.653259 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2919d823-3643-43f7-9a68-4640fdfc25c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2919d823-3643-43f7-9a68-4640fdfc25c8-plugins-config-volume\") pod \"2919d823-3643-43f7-9a68-4640fdfc25c8\" (UID: \"2919d823-3643-43f7-9a68-4640fdfc25c8\") " Nov 01 01:03:43.653506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.653391 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2919d823-3643-43f7-9a68-4640fdfc25c8-kube-api-access-6vtn6\" (UniqueName: \"kubernetes.io/projected/2919d823-3643-43f7-9a68-4640fdfc25c8-kube-api-access-6vtn6\") pod \"2919d823-3643-43f7-9a68-4640fdfc25c8\" (UID: \"2919d823-3643-43f7-9a68-4640fdfc25c8\") " Nov 01 01:03:43.653635 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.653612 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2919d823-3643-43f7-9a68-4640fdfc25c8-plugins-config-volume" pod "2919d823-3643-43f7-9a68-4640fdfc25c8" (UID: "2919d823-3643-43f7-9a68-4640fdfc25c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:43.653929 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.653907 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcqt\" (UniqueName: \"kubernetes.io/projected/eac98067-e982-45a1-8b6a-022e2e331657-kube-api-access-nwcqt\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:43.653929 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.653926 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2919d823-3643-43f7-9a68-4640fdfc25c8-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:43.655786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.655763 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2919d823-3643-43f7-9a68-4640fdfc25c8-kube-api-access-6vtn6" pod "2919d823-3643-43f7-9a68-4640fdfc25c8" (UID: "2919d823-3643-43f7-9a68-4640fdfc25c8"). InnerVolumeSpecName "kube-api-access-6vtn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:43.755876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:43.755780 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vtn6\" (UniqueName: \"kubernetes.io/projected/2919d823-3643-43f7-9a68-4640fdfc25c8-kube-api-access-6vtn6\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:44.074396 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.074349 6036 generic.go:356] "Generic (PLEG): container finished" podID="2919d823-3643-43f7-9a68-4640fdfc25c8" containerID="f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90" exitCode=0 Nov 01 01:03:44.074396 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.074391 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" Nov 01 01:03:44.074650 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.074430 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" event={"ID":"2919d823-3643-43f7-9a68-4640fdfc25c8","Type":"ContainerDied","Data":"f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90"} Nov 01 01:03:44.074650 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.074484 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d" event={"ID":"2919d823-3643-43f7-9a68-4640fdfc25c8","Type":"ContainerDied","Data":"62923b1536152ee4542c4817ad187cf159feda0a471a6290270033142261da4f"} Nov 01 01:03:44.074650 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.074509 6036 scope.go:122] "RemoveContainer" containerID="f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90" Nov 01 01:03:44.076008 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.075954 6036 generic.go:356] "Generic (PLEG): container finished" podID="eac98067-e982-45a1-8b6a-022e2e331657" containerID="b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915" exitCode=0 Nov 01 01:03:44.076175 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.076094 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" event={"ID":"eac98067-e982-45a1-8b6a-022e2e331657","Type":"ContainerDied","Data":"b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915"} Nov 01 01:03:44.076251 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.076171 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" event={"ID":"eac98067-e982-45a1-8b6a-022e2e331657","Type":"ContainerDied","Data":"637d415517787e2e934bd964066db7ad989f413aa570c1bebdf66783005b1d17"} Nov 01 01:03:44.076251 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.075995 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj" Nov 01 01:03:44.085444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.085422 6036 scope.go:122] "RemoveContainer" containerID="f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90" Nov 01 01:03:44.085748 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:44.085730 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90\": container with ID starting with f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90 not found: ID does not exist" containerID="f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90" Nov 01 01:03:44.085837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.085760 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90"} err="failed to get container status \"f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90\": rpc error: code = NotFound desc = could not find container \"f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90\": container with ID starting with f5e48f1431fa036dfd59343b784e4ca874242d56658e9f476bc0c53b5a01ba90 not found: ID does not exist" Nov 01 01:03:44.085837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.085785 6036 scope.go:122] "RemoveContainer" containerID="b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915" Nov 01 01:03:44.095444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.095426 6036 scope.go:122] "RemoveContainer" containerID="b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915" Nov 01 01:03:44.095723 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:44.095704 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915\": container with ID starting with b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915 not found: ID does not exist" containerID="b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915" Nov 01 01:03:44.095765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.095733 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915"} err="failed to get container status \"b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915\": rpc error: code = NotFound desc = could not find container \"b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915\": container with ID starting with b0629f015ce175801ffd09a4dca7c60fc30a4817a70c95089e433aa8834e8915 not found: ID does not exist" Nov 01 01:03:44.243610 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.243575 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d"] Nov 01 01:03:44.250355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.250329 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ba5b10-84d8d4859-lcm4d"] Nov 01 01:03:44.368828 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.368755 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj"] Nov 01 01:03:44.377696 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:44.377665 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ba5b10-58f447dc86-cnhsj"] Nov 01 01:03:45.251891 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:45.251858 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2919d823-3643-43f7-9a68-4640fdfc25c8" path="/var/lib/kubelet/pods/2919d823-3643-43f7-9a68-4640fdfc25c8/volumes" Nov 01 01:03:45.252369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:45.252354 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac98067-e982-45a1-8b6a-022e2e331657" path="/var/lib/kubelet/pods/eac98067-e982-45a1-8b6a-022e2e331657/volumes" Nov 01 01:03:45.476744 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:45.476704 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g"] Nov 01 01:03:45.905878 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:45.905841 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:46.119788 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:46.119754 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:46.438125 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:46.438088 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:49.457035 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:49.456998 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:49.506057 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:49.506018 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:03:49.658868 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:49.658833 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:50.477244 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.477203 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" podUID="4823c102-4b66-4381-a596-6e330e729e77" containerName="payload-processing" containerID="cri-o://c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5" gracePeriod=25 Nov 01 01:03:50.632120 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:03:50.632081 6036 logging.go:55] [core] [Channel #1982 SubChannel #1983]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.105:50052", ServerName: "10.134.0.105:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.105:50052: connect: connection refused" Nov 01 01:03:50.715109 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.715061 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:50.758403 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.758354 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 01:03:50.855904 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.855869 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4823c102-4b66-4381-a596-6e330e729e77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4823c102-4b66-4381-a596-6e330e729e77-plugins-config-volume\") pod \"4823c102-4b66-4381-a596-6e330e729e77\" (UID: \"4823c102-4b66-4381-a596-6e330e729e77\") " Nov 01 01:03:50.856141 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.856128 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4823c102-4b66-4381-a596-6e330e729e77-kube-api-access-ptds4\" (UniqueName: \"kubernetes.io/projected/4823c102-4b66-4381-a596-6e330e729e77-kube-api-access-ptds4\") pod \"4823c102-4b66-4381-a596-6e330e729e77\" (UID: \"4823c102-4b66-4381-a596-6e330e729e77\") " Nov 01 01:03:50.856312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.856271 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4823c102-4b66-4381-a596-6e330e729e77-plugins-config-volume" pod "4823c102-4b66-4381-a596-6e330e729e77" (UID: "4823c102-4b66-4381-a596-6e330e729e77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:50.856689 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.856667 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4823c102-4b66-4381-a596-6e330e729e77-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:50.858487 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.858467 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4823c102-4b66-4381-a596-6e330e729e77-kube-api-access-ptds4" pod "4823c102-4b66-4381-a596-6e330e729e77" (UID: "4823c102-4b66-4381-a596-6e330e729e77"). InnerVolumeSpecName "kube-api-access-ptds4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:50.957734 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:50.957703 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptds4\" (UniqueName: \"kubernetes.io/projected/4823c102-4b66-4381-a596-6e330e729e77-kube-api-access-ptds4\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:51.129984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.129950 6036 generic.go:356] "Generic (PLEG): container finished" podID="4823c102-4b66-4381-a596-6e330e729e77" containerID="c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5" exitCode=0 Nov 01 01:03:51.130203 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.130017 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" Nov 01 01:03:51.130203 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.130039 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" event={"ID":"4823c102-4b66-4381-a596-6e330e729e77","Type":"ContainerDied","Data":"c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5"} Nov 01 01:03:51.130203 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.130134 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" event={"ID":"4823c102-4b66-4381-a596-6e330e729e77","Type":"ContainerDied","Data":"691277955bdf3f9a7d41e2653e5e07d3824a1073724031551b893210d365b5f0"} Nov 01 01:03:51.130203 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.130163 6036 scope.go:122] "RemoveContainer" containerID="c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5" Nov 01 01:03:51.143972 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.143953 6036 scope.go:122] "RemoveContainer" containerID="c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5" Nov 01 01:03:51.144266 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:51.144247 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5\": container with ID starting with c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5 not found: ID does not exist" containerID="c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5" Nov 01 01:03:51.144358 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.144275 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5"} err="failed to get container status \"c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5\": rpc error: code = NotFound desc = could not find container \"c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5\": container with ID starting with c2c8f3a7cc845260ed6d801fdc2362e44eb5c0f38efda7fd280c1fb60ea59bc5 not found: ID does not exist" Nov 01 01:03:51.300763 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.300731 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g"] Nov 01 01:03:51.303545 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.303523 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g"] Nov 01 01:03:51.632661 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:51.632610 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc241b-6744679db-4lk6g" podUID="4823c102-4b66-4381-a596-6e330e729e77" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.105:50052\" within 1s: context deadline exceeded" Nov 01 01:03:52.359454 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:52.359421 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:52.637751 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:52.637654 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:52.856099 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:52.856059 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:53.253748 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:53.253712 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4823c102-4b66-4381-a596-6e330e729e77" path="/var/lib/kubelet/pods/4823c102-4b66-4381-a596-6e330e729e77/volumes" Nov 01 01:03:53.320456 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:53.320422 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:55.674428 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:55.674395 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:55.883603 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:55.883569 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:57.151658 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.151623 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5"] Nov 01 01:03:57.152184 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.151882 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" podUID="4510bd21-cf38-42c7-9fe2-cf19a021af4d" containerName="maas-api" containerID="cri-o://69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a" gracePeriod=30 Nov 01 01:03:57.312113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.312081 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:57.441550 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.441522 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 01:03:57.557188 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.557158 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4510bd21-cf38-42c7-9fe2-cf19a021af4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4510bd21-cf38-42c7-9fe2-cf19a021af4d-openshift-service-ca\") pod \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " Nov 01 01:03:57.557427 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.557277 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls\") pod \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " Nov 01 01:03:57.557427 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.557351 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-metrics-certs\") pod \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " Nov 01 01:03:57.557537 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.557429 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4510bd21-cf38-42c7-9fe2-cf19a021af4d-kube-api-access-g4srh\" (UniqueName: \"kubernetes.io/projected/4510bd21-cf38-42c7-9fe2-cf19a021af4d-kube-api-access-g4srh\") pod \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\" (UID: \"4510bd21-cf38-42c7-9fe2-cf19a021af4d\") " Nov 01 01:03:57.557938 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.557853 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4510bd21-cf38-42c7-9fe2-cf19a021af4d-openshift-service-ca" pod "4510bd21-cf38-42c7-9fe2-cf19a021af4d" (UID: "4510bd21-cf38-42c7-9fe2-cf19a021af4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:57.558375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.558133 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4510bd21-cf38-42c7-9fe2-cf19a021af4d-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:57.560456 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.560412 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls" pod "4510bd21-cf38-42c7-9fe2-cf19a021af4d" (UID: "4510bd21-cf38-42c7-9fe2-cf19a021af4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:03:57.578440 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.577404 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-metrics-certs" pod "4510bd21-cf38-42c7-9fe2-cf19a021af4d" (UID: "4510bd21-cf38-42c7-9fe2-cf19a021af4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:03:57.626578 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.610723 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4510bd21-cf38-42c7-9fe2-cf19a021af4d-kube-api-access-g4srh" pod "4510bd21-cf38-42c7-9fe2-cf19a021af4d" (UID: "4510bd21-cf38-42c7-9fe2-cf19a021af4d"). InnerVolumeSpecName "kube-api-access-g4srh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:57.659118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.659035 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:57.659118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.659065 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4510bd21-cf38-42c7-9fe2-cf19a021af4d-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:57.659118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:57.659082 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4srh\" (UniqueName: \"kubernetes.io/projected/4510bd21-cf38-42c7-9fe2-cf19a021af4d-kube-api-access-g4srh\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:58.196319 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.196264 6036 generic.go:356] "Generic (PLEG): container finished" podID="4510bd21-cf38-42c7-9fe2-cf19a021af4d" containerID="69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a" exitCode=0 Nov 01 01:03:58.196837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.196355 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" event={"ID":"4510bd21-cf38-42c7-9fe2-cf19a021af4d","Type":"ContainerDied","Data":"69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a"} Nov 01 01:03:58.196837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.196403 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" event={"ID":"4510bd21-cf38-42c7-9fe2-cf19a021af4d","Type":"ContainerDied","Data":"272d94ee461072f34ac4ca1370e2e041f474cc52cd126893e262d88f549d2988"} Nov 01 01:03:58.196837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.196429 6036 scope.go:122] "RemoveContainer" containerID="69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a" Nov 01 01:03:58.196837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.196561 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5" Nov 01 01:03:58.214598 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.214578 6036 scope.go:122] "RemoveContainer" containerID="69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a" Nov 01 01:03:58.214926 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:03:58.214906 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a\": container with ID starting with 69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a not found: ID does not exist" containerID="69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a" Nov 01 01:03:58.215014 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.214934 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a"} err="failed to get container status \"69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a\": rpc error: code = NotFound desc = could not find container \"69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a\": container with ID starting with 69b3df4e0d0caac505333824cbf06de8d46908bcee8de5fb553ab62d838ec81a not found: ID does not exist" Nov 01 01:03:58.366824 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.366791 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5"] Nov 01 01:03:58.417946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.417912 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b112c158-799d88785c-vsdc5"] Nov 01 01:03:58.914151 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:58.914114 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:59.142214 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:59.142179 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:03:59.251847 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:59.251769 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4510bd21-cf38-42c7-9fe2-cf19a021af4d" path="/var/lib/kubelet/pods/4510bd21-cf38-42c7-9fe2-cf19a021af4d/volumes" Nov 01 01:03:59.356897 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:03:59.356830 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:00.123444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:00.123406 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:00.340473 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:00.340436 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:00.565137 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:00.565102 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:01.356697 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.356661 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:01.820439 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.820400 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d"] Nov 01 01:04:01.821025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821011 6036 cpu_manager.go:414] "containerMap: removing container" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" Nov 01 01:04:01.821076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821026 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" Nov 01 01:04:01.821076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821044 6036 cpu_manager.go:414] "containerMap: removing container" podUID="599bb32b-a4fd-4cb6-9787-4ae7885dc770" containerName="payload-pre-processing" Nov 01 01:04:01.821076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821049 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599bb32b-a4fd-4cb6-9787-4ae7885dc770" containerName="payload-pre-processing" Nov 01 01:04:01.821076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821060 6036 cpu_manager.go:414] "containerMap: removing container" podUID="eac98067-e982-45a1-8b6a-022e2e331657" containerName="payload-processing" Nov 01 01:04:01.821076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821065 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac98067-e982-45a1-8b6a-022e2e331657" containerName="payload-processing" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821080 6036 cpu_manager.go:414] "containerMap: removing container" podUID="db4381d5-90f1-407d-9130-7ce9910ce004" containerName="maas-api" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821085 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4381d5-90f1-407d-9130-7ce9910ce004" containerName="maas-api" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821099 6036 cpu_manager.go:414] "containerMap: removing container" podUID="4510bd21-cf38-42c7-9fe2-cf19a021af4d" containerName="maas-api" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821104 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4510bd21-cf38-42c7-9fe2-cf19a021af4d" containerName="maas-api" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821116 6036 cpu_manager.go:414] "containerMap: removing container" podUID="b0c228fd-2e0f-4c82-a177-1bae31193c6a" containerName="discovery" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821121 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c228fd-2e0f-4c82-a177-1bae31193c6a" containerName="discovery" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821134 6036 cpu_manager.go:414] "containerMap: removing container" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821139 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821147 6036 cpu_manager.go:414] "containerMap: removing container" podUID="8ee7203e-1654-44fc-9f9a-5c3d5d7db131" containerName="payload-processing" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821151 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee7203e-1654-44fc-9f9a-5c3d5d7db131" containerName="payload-processing" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821162 6036 cpu_manager.go:414] "containerMap: removing container" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821166 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821175 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821179 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821186 6036 cpu_manager.go:414] "containerMap: removing container" podUID="4823c102-4b66-4381-a596-6e330e729e77" containerName="payload-processing" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821190 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4823c102-4b66-4381-a596-6e330e729e77" containerName="payload-processing" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821200 6036 cpu_manager.go:414] "containerMap: removing container" podUID="82a400a2-938a-4ff3-9c1e-5c0f51baf810" containerName="payload-pre-processing" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821204 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a400a2-938a-4ff3-9c1e-5c0f51baf810" containerName="payload-pre-processing" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821213 6036 cpu_manager.go:414] "containerMap: removing container" podUID="2919d823-3643-43f7-9a68-4640fdfc25c8" containerName="payload-pre-processing" Nov 01 01:04:01.821233 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821217 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2919d823-3643-43f7-9a68-4640fdfc25c8" containerName="payload-pre-processing" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821322 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee7203e-1654-44fc-9f9a-5c3d5d7db131" containerName="payload-processing" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821331 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e00ff82-0096-4553-9f58-43b412f61f30" containerName="istio-proxy" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821338 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9027416f-3c14-453c-ae29-81c0c7819e2d" containerName="istio-proxy" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821343 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821349 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4823c102-4b66-4381-a596-6e330e729e77" containerName="payload-processing" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821355 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a400a2-938a-4ff3-9c1e-5c0f51baf810" containerName="payload-pre-processing" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821362 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2919d823-3643-43f7-9a68-4640fdfc25c8" containerName="payload-pre-processing" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821370 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821376 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="599bb32b-a4fd-4cb6-9787-4ae7885dc770" containerName="payload-pre-processing" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821381 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac98067-e982-45a1-8b6a-022e2e331657" containerName="payload-processing" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821388 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4381d5-90f1-407d-9130-7ce9910ce004" containerName="maas-api" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821396 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4510bd21-cf38-42c7-9fe2-cf19a021af4d" containerName="maas-api" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821402 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c228fd-2e0f-4c82-a177-1bae31193c6a" containerName="discovery" Nov 01 01:04:01.821856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.821408 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf10d2f9-c3ea-43e1-bb2e-9345697fbccc" containerName="istio-proxy" Nov 01 01:04:01.824766 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.824749 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.933636 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.918257 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-603e8c-openshift-default-dockercfg-ql7ln\"" Nov 01 01:04:01.940679 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.940644 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d"] Nov 01 01:04:01.943298 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.943251 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-podinfo\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.947934 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.947907 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dc99\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-kube-api-access-2dc99\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.948243 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.948224 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istiod-ca-cert\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.948506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.948470 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-data\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.948647 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.948631 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-socket\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.948834 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.948820 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-token\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.948998 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.948985 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-certs\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.949250 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.949155 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-envoy\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.949479 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.949462 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-credential-socket\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:01.973164 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:01.973124 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:02.051466 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051433 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-credential-socket\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.051660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051488 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-podinfo\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.051660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051527 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dc99\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-kube-api-access-2dc99\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.051660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051583 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istiod-ca-cert\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.051660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051623 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-data\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.051660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051642 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-socket\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.051932 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051705 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-token\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.051932 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051786 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-certs\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.051932 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051814 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-envoy\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.052099 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.051941 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-credential-socket\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.052099 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.052050 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-data\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.052196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.052126 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-socket\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.052196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.052187 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-certs\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.052547 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.052467 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istiod-ca-cert\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.054168 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.054145 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-envoy\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.054490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.054474 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-podinfo\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.140390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.140311 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-token\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.147436 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.147407 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dc99\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-kube-api-access-2dc99\") pod \"e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.394152 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.394062 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:02.439378 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.439341 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:02.881222 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.881189 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d"] Nov 01 01:04:02.881578 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:04:02.881547 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244bd1e4_d0c1_45b9_a8bc_ca172ba181a6.slice/crio-77915e6733efd7d82462088ecfc79c12a064c937ad332f5d2c7fc0d978d8a6d3 WatchSource:0}: Error finding container 77915e6733efd7d82462088ecfc79c12a064c937ad332f5d2c7fc0d978d8a6d3: Status 404 returned error can't find the container with id 77915e6733efd7d82462088ecfc79c12a064c937ad332f5d2c7fc0d978d8a6d3 Nov 01 01:04:02.884472 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.884437 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:04:02.884570 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.884502 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:04:02.884570 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:02.884549 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:04:03.238386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:03.238278 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" event={"ID":"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6","Type":"ContainerStarted","Data":"807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181"} Nov 01 01:04:03.238386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:03.238361 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" event={"ID":"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6","Type":"ContainerStarted","Data":"77915e6733efd7d82462088ecfc79c12a064c937ad332f5d2c7fc0d978d8a6d3"} Nov 01 01:04:03.363636 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:03.363587 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" podStartSLOduration=2.363574792 podStartE2EDuration="2.363574792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:04:03.362804726 +0000 UTC m=+2388.238025742" watchObservedRunningTime="2026-11-01 01:04:03.363574792 +0000 UTC m=+2388.238795806" Nov 01 01:04:03.440401 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:03.440358 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:03.446736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:03.446709 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:03.550331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:03.543782 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:04.245769 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:04.245737 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:04.246671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:04.246652 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:04:04.907659 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:04.907624 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:06.418927 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:06.418894 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:08.041616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:08.041583 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:09.534224 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.532761 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:09.739126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.739090 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:09.789685 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.789596 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj"] Nov 01 01:04:09.790364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.790342 6036 cpu_manager.go:414] "containerMap: removing container" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" Nov 01 01:04:09.790364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.790363 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c93b96-6c9c-4cbc-901a-dc98de25ec1a" containerName="istio-proxy" Nov 01 01:04:09.794078 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.794056 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:09.851136 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.851103 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj"] Nov 01 01:04:09.871145 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.871108 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050fffbe-b376-4cbc-ab7f-8b1c76349986-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj\" (UID: \"050fffbe-b376-4cbc-ab7f-8b1c76349986\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:09.871370 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.871336 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwrp\" (UniqueName: \"kubernetes.io/projected/050fffbe-b376-4cbc-ab7f-8b1c76349986-kube-api-access-7lwrp\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj\" (UID: \"050fffbe-b376-4cbc-ab7f-8b1c76349986\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:09.924668 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.924635 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b112c158-dockercfg-fgv8q\"" Nov 01 01:04:09.925131 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.925109 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b112c158\"" Nov 01 01:04:09.973857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.973825 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwrp\" (UniqueName: \"kubernetes.io/projected/050fffbe-b376-4cbc-ab7f-8b1c76349986-kube-api-access-7lwrp\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj\" (UID: \"050fffbe-b376-4cbc-ab7f-8b1c76349986\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:09.974126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.973977 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050fffbe-b376-4cbc-ab7f-8b1c76349986-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj\" (UID: \"050fffbe-b376-4cbc-ab7f-8b1c76349986\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:09.974789 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.974762 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050fffbe-b376-4cbc-ab7f-8b1c76349986-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj\" (UID: \"050fffbe-b376-4cbc-ab7f-8b1c76349986\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:09.980876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.980845 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj"] Nov 01 01:04:09.985477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:09.985455 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:10.018932 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.018899 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj"] Nov 01 01:04:10.039001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.038963 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwrp\" (UniqueName: \"kubernetes.io/projected/050fffbe-b376-4cbc-ab7f-8b1c76349986-kube-api-access-7lwrp\") pod \"payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj\" (UID: \"050fffbe-b376-4cbc-ab7f-8b1c76349986\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:10.075837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.075755 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-plugins-config-volume\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj\" (UID: \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:10.075984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.075953 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ddk\" (UniqueName: \"kubernetes.io/projected/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-kube-api-access-q8ddk\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj\" (UID: \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:10.108734 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.108707 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:10.166951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.166907 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:10.177727 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.177693 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ddk\" (UniqueName: \"kubernetes.io/projected/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-kube-api-access-q8ddk\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj\" (UID: \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:10.177916 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.177886 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-plugins-config-volume\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj\" (UID: \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:10.178864 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.178839 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-plugins-config-volume\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj\" (UID: \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:10.259328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.259259 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ddk\" (UniqueName: \"kubernetes.io/projected/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-kube-api-access-q8ddk\") pod \"payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj\" (UID: \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\") " pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:10.304695 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.304662 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:10.525951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.525918 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj"] Nov 01 01:04:10.526539 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:04:10.526512 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050fffbe_b376_4cbc_ab7f_8b1c76349986.slice/crio-ccd359894fcfd30657efe5d90ae902602003b78f8d3f900df2dc17045b4912b8 WatchSource:0}: Error finding container ccd359894fcfd30657efe5d90ae902602003b78f8d3f900df2dc17045b4912b8: Status 404 returned error can't find the container with id ccd359894fcfd30657efe5d90ae902602003b78f8d3f900df2dc17045b4912b8 Nov 01 01:04:10.687571 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:10.687540 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj"] Nov 01 01:04:10.688496 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:04:10.688470 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fb595d_d9fe_40e8_9f09_8eea308bf3d7.slice/crio-364bfdda3e10ad27cb4e0f0a7e25f2b727878787d1b2a41dfd330792d8c679b0 WatchSource:0}: Error finding container 364bfdda3e10ad27cb4e0f0a7e25f2b727878787d1b2a41dfd330792d8c679b0: Status 404 returned error can't find the container with id 364bfdda3e10ad27cb4e0f0a7e25f2b727878787d1b2a41dfd330792d8c679b0 Nov 01 01:04:11.296663 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:11.296628 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" event={"ID":"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7","Type":"ContainerStarted","Data":"894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6"} Nov 01 01:04:11.296880 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:11.296690 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" event={"ID":"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7","Type":"ContainerStarted","Data":"364bfdda3e10ad27cb4e0f0a7e25f2b727878787d1b2a41dfd330792d8c679b0"} Nov 01 01:04:11.296880 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:11.296716 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:11.298328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:11.298263 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" event={"ID":"050fffbe-b376-4cbc-ab7f-8b1c76349986","Type":"ContainerStarted","Data":"6bc22843b42e5c555a501c241ecff82193bebd59510cb87ae7bc86be3b3acca5"} Nov 01 01:04:11.298459 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:11.298350 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" event={"ID":"050fffbe-b376-4cbc-ab7f-8b1c76349986","Type":"ContainerStarted","Data":"ccd359894fcfd30657efe5d90ae902602003b78f8d3f900df2dc17045b4912b8"} Nov 01 01:04:11.298459 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:11.298377 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:11.418925 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:11.418878 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" podStartSLOduration=2.418864501 podStartE2EDuration="2.418864501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:04:11.415273769 +0000 UTC m=+2396.290494786" watchObservedRunningTime="2026-11-01 01:04:11.418864501 +0000 UTC m=+2396.294085560" Nov 01 01:04:11.540104 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:11.539899 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" podStartSLOduration=2.5398758519999998 podStartE2EDuration="2.539875852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:04:11.536592417 +0000 UTC m=+2396.411813437" watchObservedRunningTime="2026-11-01 01:04:11.539875852 +0000 UTC m=+2396.415096868" Nov 01 01:04:14.288118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.288091 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:14.497785 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.497754 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:14.636525 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.636439 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b"] Nov 01 01:04:14.641349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.641324 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:14.698078 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.698045 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b"] Nov 01 01:04:14.747345 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.746645 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-603e8c-dockercfg-gxrsb\"" Nov 01 01:04:14.747345 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.746729 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-603e8c\"" Nov 01 01:04:14.765673 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.755636 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b\" (UID: \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:14.765673 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.756025 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tgk\" (UniqueName: \"kubernetes.io/projected/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-kube-api-access-22tgk\") pod \"payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b\" (UID: \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:14.824478 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.824431 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r"] Nov 01 01:04:14.830666 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.830618 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:14.852274 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.850319 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r"] Nov 01 01:04:14.860384 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.858976 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tgk\" (UniqueName: \"kubernetes.io/projected/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-kube-api-access-22tgk\") pod \"payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b\" (UID: \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:14.860384 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.859443 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b\" (UID: \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:14.861719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.861682 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b\" (UID: \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:14.921494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.921417 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tgk\" (UniqueName: \"kubernetes.io/projected/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-kube-api-access-22tgk\") pod \"payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b\" (UID: \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:14.957591 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.957562 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:14.961123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.961099 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7cde15-0d3d-47b3-bf40-3256cfced049-plugins-config-volume\") pod \"payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r\" (UID: \"af7cde15-0d3d-47b3-bf40-3256cfced049\") " pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:14.961186 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:14.961172 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rtd\" (UniqueName: \"kubernetes.io/projected/af7cde15-0d3d-47b3-bf40-3256cfced049-kube-api-access-85rtd\") pod \"payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r\" (UID: \"af7cde15-0d3d-47b3-bf40-3256cfced049\") " pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:15.063345 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.063175 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rtd\" (UniqueName: \"kubernetes.io/projected/af7cde15-0d3d-47b3-bf40-3256cfced049-kube-api-access-85rtd\") pod \"payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r\" (UID: \"af7cde15-0d3d-47b3-bf40-3256cfced049\") " pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:15.063549 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.063516 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7cde15-0d3d-47b3-bf40-3256cfced049-plugins-config-volume\") pod \"payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r\" (UID: \"af7cde15-0d3d-47b3-bf40-3256cfced049\") " pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:15.066107 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.065034 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7cde15-0d3d-47b3-bf40-3256cfced049-plugins-config-volume\") pod \"payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r\" (UID: \"af7cde15-0d3d-47b3-bf40-3256cfced049\") " pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:15.125553 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.125519 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rtd\" (UniqueName: \"kubernetes.io/projected/af7cde15-0d3d-47b3-bf40-3256cfced049-kube-api-access-85rtd\") pod \"payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r\" (UID: \"af7cde15-0d3d-47b3-bf40-3256cfced049\") " pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:15.164401 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.164367 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:15.340077 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.340055 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b"] Nov 01 01:04:15.341053 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:04:15.341025 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode535ab08_281e_4d91_b7fa_2647a4f8c7e8.slice/crio-c17837963341d615aa67281d03c132abddac42d6377f36532df7582c4320512d WatchSource:0}: Error finding container c17837963341d615aa67281d03c132abddac42d6377f36532df7582c4320512d: Status 404 returned error can't find the container with id c17837963341d615aa67281d03c132abddac42d6377f36532df7582c4320512d Nov 01 01:04:15.350809 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.350781 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" event={"ID":"e535ab08-281e-4d91-b7fa-2647a4f8c7e8","Type":"ContainerStarted","Data":"c17837963341d615aa67281d03c132abddac42d6377f36532df7582c4320512d"} Nov 01 01:04:15.546641 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.546611 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r"] Nov 01 01:04:15.547135 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:04:15.547110 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7cde15_0d3d_47b3_bf40_3256cfced049.slice/crio-70ad2ca86f07115ac35c2da7bba98add60a4fb1e87e2b85aa53ff6aebc35ea61 WatchSource:0}: Error finding container 70ad2ca86f07115ac35c2da7bba98add60a4fb1e87e2b85aa53ff6aebc35ea61: Status 404 returned error can't find the container with id 70ad2ca86f07115ac35c2da7bba98add60a4fb1e87e2b85aa53ff6aebc35ea61 Nov 01 01:04:15.592406 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.592365 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:15.812653 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:15.812621 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:16.024798 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:16.024761 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:16.359817 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:16.359783 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" event={"ID":"e535ab08-281e-4d91-b7fa-2647a4f8c7e8","Type":"ContainerStarted","Data":"76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1"} Nov 01 01:04:16.360321 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:16.359981 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:16.361564 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:16.361538 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" event={"ID":"af7cde15-0d3d-47b3-bf40-3256cfced049","Type":"ContainerStarted","Data":"f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c"} Nov 01 01:04:16.361671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:16.361584 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" event={"ID":"af7cde15-0d3d-47b3-bf40-3256cfced049","Type":"ContainerStarted","Data":"70ad2ca86f07115ac35c2da7bba98add60a4fb1e87e2b85aa53ff6aebc35ea61"} Nov 01 01:04:16.361671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:16.361614 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:16.477913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:16.477862 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" podStartSLOduration=2.477848688 podStartE2EDuration="2.477848688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:04:16.477374125 +0000 UTC m=+2401.352595143" watchObservedRunningTime="2026-11-01 01:04:16.477848688 +0000 UTC m=+2401.353069702" Nov 01 01:04:16.604053 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:16.602245 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" podStartSLOduration=2.60222524 podStartE2EDuration="2.60222524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:04:16.597460107 +0000 UTC m=+2401.472681125" watchObservedRunningTime="2026-11-01 01:04:16.60222524 +0000 UTC m=+2401.477446256" Nov 01 01:04:19.507618 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:04:19.507581 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:04:21.083011 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:21.082958 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:22.320708 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:22.320681 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:04:22.321093 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:22.320747 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:04:26.292408 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:26.292370 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:27.372624 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:27.372595 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:04:27.373012 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:27.372733 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:04:31.678644 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:31.678609 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:34.317608 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:34.317574 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:36.814604 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:36.811644 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:39.904563 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:39.904528 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:40.121117 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:40.121083 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:41.602499 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:41.602263 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:45.825987 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:45.825952 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:47.005390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:47.005349 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:47.379854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:47.379821 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:47.676081 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:47.675987 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:48.321583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:48.321551 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:48.732971 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:48.732887 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:49.056788 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:49.056754 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:49.265799 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:49.265766 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:49.507088 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:04:49.506995 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:04:49.701069 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:49.701037 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:53.647352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:53.647278 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:53.894395 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:53.891450 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:54.113154 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:54.113113 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:55.010608 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:55.010565 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:55.239088 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:55.239050 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:55.291584 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:55.291550 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9"] Nov 01 01:04:55.291845 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:55.291796 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" containerID="cri-o://9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54" gracePeriod=30 Nov 01 01:04:55.444966 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:55.444929 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:55.654034 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:55.653942 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:55.921580 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:55.921488 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:56.134830 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:56.134794 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:56.533559 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:56.533520 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:56.745511 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:56.745467 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:04:56.959762 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:04:56.959679 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:00.590619 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.590591 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 01:05:00.682686 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.682652 6036 generic.go:356] "Generic (PLEG): container finished" podID="4f402208-c342-4429-8b0b-3259e410455d" containerID="9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54" exitCode=0 Nov 01 01:05:00.682869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.682714 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" Nov 01 01:05:00.682869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.682741 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" event={"ID":"4f402208-c342-4429-8b0b-3259e410455d","Type":"ContainerDied","Data":"9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54"} Nov 01 01:05:00.682869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.682807 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9" event={"ID":"4f402208-c342-4429-8b0b-3259e410455d","Type":"ContainerDied","Data":"f292509764265523cdf193c87f2c90a0060994f46db70ed95974f52a48f05e33"} Nov 01 01:05:00.682869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.682833 6036 scope.go:122] "RemoveContainer" containerID="9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54" Nov 01 01:05:00.686382 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686359 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-certs\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.686522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686412 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-istio-token\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-istio-token\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.686522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686448 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f402208-c342-4429-8b0b-3259e410455d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f402208-c342-4429-8b0b-3259e410455d-istio-podinfo\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.686522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686504 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-kube-api-access-fklww\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-kube-api-access-fklww\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.686698 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686549 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-data\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.686698 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686678 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f402208-c342-4429-8b0b-3259e410455d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f402208-c342-4429-8b0b-3259e410455d-istiod-ca-cert\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.686857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686720 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-envoy\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.686857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686781 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-credential-socket\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.686857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686831 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-data" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:00.686857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686841 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-certs" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:00.687084 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.686872 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-socket\") pod \"4f402208-c342-4429-8b0b-3259e410455d\" (UID: \"4f402208-c342-4429-8b0b-3259e410455d\") " Nov 01 01:05:00.688371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.687272 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f402208-c342-4429-8b0b-3259e410455d-istiod-ca-cert" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:00.688371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.687650 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.688371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.687674 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.688371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.687689 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f402208-c342-4429-8b0b-3259e410455d-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.688371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.687668 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-socket" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:00.690345 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.689234 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f402208-c342-4429-8b0b-3259e410455d-istio-podinfo" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:05:00.690527 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.690499 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-credential-socket" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:00.690649 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.690589 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-kube-api-access-fklww" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "kube-api-access-fklww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:00.692727 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.692696 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-istio-token" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:00.692812 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.692745 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-envoy" pod "4f402208-c342-4429-8b0b-3259e410455d" (UID: "4f402208-c342-4429-8b0b-3259e410455d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:00.697561 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.697455 6036 scope.go:122] "RemoveContainer" containerID="b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea" Nov 01 01:05:00.723359 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.723335 6036 scope.go:122] "RemoveContainer" containerID="9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54" Nov 01 01:05:00.723721 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:05:00.723697 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54\": container with ID starting with 9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54 not found: ID does not exist" containerID="9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54" Nov 01 01:05:00.723826 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.723735 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54"} err="failed to get container status \"9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54\": rpc error: code = NotFound desc = could not find container \"9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54\": container with ID starting with 9454a448fcc098bcf537d4c4ce41861eefee17c13083338da2e434c2378a2b54 not found: ID does not exist" Nov 01 01:05:00.723826 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.723760 6036 scope.go:122] "RemoveContainer" containerID="b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea" Nov 01 01:05:00.724056 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:05:00.724039 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea\": container with ID starting with b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea not found: ID does not exist" containerID="b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea" Nov 01 01:05:00.724119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.724065 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea"} err="failed to get container status \"b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea\": rpc error: code = NotFound desc = could not find container \"b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea\": container with ID starting with b01a060537b5f71ff7ef09b26da8132e856fcf223efc0a5856a469844772ebea not found: ID does not exist" Nov 01 01:05:00.790109 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.790071 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.790109 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.790111 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f402208-c342-4429-8b0b-3259e410455d-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.790369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.790129 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fklww\" (UniqueName: \"kubernetes.io/projected/4f402208-c342-4429-8b0b-3259e410455d-kube-api-access-fklww\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.790369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.790143 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.790369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.790157 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.790369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:00.790170 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f402208-c342-4429-8b0b-3259e410455d-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:01.112163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:01.112108 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9"] Nov 01 01:05:01.116073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:01.116043 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ba5b10-openshift-default-6867947569-6dck9"] Nov 01 01:05:01.219004 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:01.218963 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:01.253156 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:01.253076 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f402208-c342-4429-8b0b-3259e410455d" path="/var/lib/kubelet/pods/4f402208-c342-4429-8b0b-3259e410455d/volumes" Nov 01 01:05:02.956059 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:02.956029 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:03.183343 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:03.181833 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:03.392645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:03.392609 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:04.171558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:04.171524 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:04.973346 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:04.973311 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc"] Nov 01 01:05:05.050996 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:05.050961 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:05.080942 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:05.080911 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27"] Nov 01 01:05:05.415935 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:05.415898 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:05.636417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:05.636380 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:06.548256 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:06.548217 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:07.526004 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:07.525971 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4"] Nov 01 01:05:07.634203 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:07.634165 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f"] Nov 01 01:05:07.917191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:07.917155 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:08.127239 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:08.127208 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:09.780057 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:09.780021 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:09.973887 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:09.973845 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" podUID="91185e42-540e-4fd4-bf03-f52783251c1f" containerName="payload-processing" containerID="cri-o://ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52" gracePeriod=25 Nov 01 01:05:10.055945 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.055910 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:10.082044 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.082000 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" podUID="394c5e6a-ff94-4550-b421-7686fa50097b" containerName="payload-pre-processing" containerID="cri-o://9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20" gracePeriod=25 Nov 01 01:05:10.276386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.275957 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:05:10.289830 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.289798 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91185e42-540e-4fd4-bf03-f52783251c1f-kube-api-access-76cbh\" (UniqueName: \"kubernetes.io/projected/91185e42-540e-4fd4-bf03-f52783251c1f-kube-api-access-76cbh\") pod \"91185e42-540e-4fd4-bf03-f52783251c1f\" (UID: \"91185e42-540e-4fd4-bf03-f52783251c1f\") " Nov 01 01:05:10.290050 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.290033 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91185e42-540e-4fd4-bf03-f52783251c1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91185e42-540e-4fd4-bf03-f52783251c1f-plugins-config-volume\") pod \"91185e42-540e-4fd4-bf03-f52783251c1f\" (UID: \"91185e42-540e-4fd4-bf03-f52783251c1f\") " Nov 01 01:05:10.290434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.290407 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91185e42-540e-4fd4-bf03-f52783251c1f-plugins-config-volume" pod "91185e42-540e-4fd4-bf03-f52783251c1f" (UID: "91185e42-540e-4fd4-bf03-f52783251c1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:10.290802 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.290777 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91185e42-540e-4fd4-bf03-f52783251c1f-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:10.292026 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.292006 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91185e42-540e-4fd4-bf03-f52783251c1f-kube-api-access-76cbh" pod "91185e42-540e-4fd4-bf03-f52783251c1f" (UID: "91185e42-540e-4fd4-bf03-f52783251c1f"). InnerVolumeSpecName "kube-api-access-76cbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:10.354232 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.354205 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:05:10.391534 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.391505 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394c5e6a-ff94-4550-b421-7686fa50097b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394c5e6a-ff94-4550-b421-7686fa50097b-plugins-config-volume\") pod \"394c5e6a-ff94-4550-b421-7686fa50097b\" (UID: \"394c5e6a-ff94-4550-b421-7686fa50097b\") " Nov 01 01:05:10.391696 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.391614 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394c5e6a-ff94-4550-b421-7686fa50097b-kube-api-access-gkxxj\" (UniqueName: \"kubernetes.io/projected/394c5e6a-ff94-4550-b421-7686fa50097b-kube-api-access-gkxxj\") pod \"394c5e6a-ff94-4550-b421-7686fa50097b\" (UID: \"394c5e6a-ff94-4550-b421-7686fa50097b\") " Nov 01 01:05:10.391872 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.391852 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394c5e6a-ff94-4550-b421-7686fa50097b-plugins-config-volume" pod "394c5e6a-ff94-4550-b421-7686fa50097b" (UID: "394c5e6a-ff94-4550-b421-7686fa50097b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:10.392132 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.392118 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76cbh\" (UniqueName: \"kubernetes.io/projected/91185e42-540e-4fd4-bf03-f52783251c1f-kube-api-access-76cbh\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:10.392171 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.392136 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394c5e6a-ff94-4550-b421-7686fa50097b-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:10.393809 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.393788 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394c5e6a-ff94-4550-b421-7686fa50097b-kube-api-access-gkxxj" pod "394c5e6a-ff94-4550-b421-7686fa50097b" (UID: "394c5e6a-ff94-4550-b421-7686fa50097b"). InnerVolumeSpecName "kube-api-access-gkxxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:10.430452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.430386 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:10.493113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.493079 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkxxj\" (UniqueName: \"kubernetes.io/projected/394c5e6a-ff94-4550-b421-7686fa50097b-kube-api-access-gkxxj\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:10.646680 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.646566 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:10.800992 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.800961 6036 generic.go:356] "Generic (PLEG): container finished" podID="91185e42-540e-4fd4-bf03-f52783251c1f" containerID="ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52" exitCode=0 Nov 01 01:05:10.801486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.801006 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" Nov 01 01:05:10.801486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.801048 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" event={"ID":"91185e42-540e-4fd4-bf03-f52783251c1f","Type":"ContainerDied","Data":"ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52"} Nov 01 01:05:10.801486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.801107 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc" event={"ID":"91185e42-540e-4fd4-bf03-f52783251c1f","Type":"ContainerDied","Data":"9a14a7d3179d9c1457803bc0295833a23b90c678d44bb4c4f41535a3558b74f1"} Nov 01 01:05:10.801486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.801137 6036 scope.go:122] "RemoveContainer" containerID="ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52" Nov 01 01:05:10.802950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.802923 6036 generic.go:356] "Generic (PLEG): container finished" podID="394c5e6a-ff94-4550-b421-7686fa50097b" containerID="9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20" exitCode=0 Nov 01 01:05:10.803069 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.802965 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" Nov 01 01:05:10.803069 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.802972 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" event={"ID":"394c5e6a-ff94-4550-b421-7686fa50097b","Type":"ContainerDied","Data":"9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20"} Nov 01 01:05:10.803069 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.803010 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27" event={"ID":"394c5e6a-ff94-4550-b421-7686fa50097b","Type":"ContainerDied","Data":"390d2505fa3320fa970a192eee110c1de652b6d310ed231ff47baa63b388d22b"} Nov 01 01:05:10.813812 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.813793 6036 scope.go:122] "RemoveContainer" containerID="ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52" Nov 01 01:05:10.814102 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:05:10.814085 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52\": container with ID starting with ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52 not found: ID does not exist" containerID="ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52" Nov 01 01:05:10.814149 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.814110 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52"} err="failed to get container status \"ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52\": rpc error: code = NotFound desc = could not find container \"ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52\": container with ID starting with ae0d74135f9488899904f38ddfa1486cb8e9d130507f41a274c332aff54a0c52 not found: ID does not exist" Nov 01 01:05:10.814149 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.814128 6036 scope.go:122] "RemoveContainer" containerID="9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20" Nov 01 01:05:10.823899 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.823882 6036 scope.go:122] "RemoveContainer" containerID="9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20" Nov 01 01:05:10.824156 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:05:10.824139 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20\": container with ID starting with 9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20 not found: ID does not exist" containerID="9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20" Nov 01 01:05:10.824215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.824163 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20"} err="failed to get container status \"9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20\": rpc error: code = NotFound desc = could not find container \"9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20\": container with ID starting with 9b0240834ccb6aeb36257ce8dff4c68a63c9df271abb5cad49bd78c85f8e1b20 not found: ID does not exist" Nov 01 01:05:10.970806 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:10.970697 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27"] Nov 01 01:05:11.023256 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:11.023220 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72acf4-6d7f64bd47-frg27"] Nov 01 01:05:11.087326 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:11.087277 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc"] Nov 01 01:05:11.092774 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:11.092748 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72acf4-6c9d9d8b9d-ftgnc"] Nov 01 01:05:11.253389 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:11.253267 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394c5e6a-ff94-4550-b421-7686fa50097b" path="/var/lib/kubelet/pods/394c5e6a-ff94-4550-b421-7686fa50097b/volumes" Nov 01 01:05:11.253808 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:11.253793 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91185e42-540e-4fd4-bf03-f52783251c1f" path="/var/lib/kubelet/pods/91185e42-540e-4fd4-bf03-f52783251c1f/volumes" Nov 01 01:05:11.832163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:11.832033 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:12.526458 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.526419 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" podUID="f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1" containerName="payload-processing" containerID="cri-o://2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532" gracePeriod=25 Nov 01 01:05:12.635317 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.635238 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" podUID="712175d1-b5eb-449f-baaa-8ffdd8c7fe4f" containerName="payload-pre-processing" containerID="cri-o://0826badebb08010ed764eaf2b1e3d239013ab08822e6537b9dc0bb764d24768e" gracePeriod=25 Nov 01 01:05:12.824628 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.824605 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:05:12.843699 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.843661 6036 generic.go:356] "Generic (PLEG): container finished" podID="f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1" containerID="2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532" exitCode=0 Nov 01 01:05:12.844171 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.843705 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" event={"ID":"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1","Type":"ContainerDied","Data":"2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532"} Nov 01 01:05:12.844171 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.843764 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" event={"ID":"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1","Type":"ContainerDied","Data":"0801cc7a8f024c6adbaf989cc6f3809686ee0d298b276fc729b55f6b4f561962"} Nov 01 01:05:12.844171 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.843767 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4" Nov 01 01:05:12.844171 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.843785 6036 scope.go:122] "RemoveContainer" containerID="2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532" Nov 01 01:05:12.846358 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.846333 6036 generic.go:356] "Generic (PLEG): container finished" podID="712175d1-b5eb-449f-baaa-8ffdd8c7fe4f" containerID="0826badebb08010ed764eaf2b1e3d239013ab08822e6537b9dc0bb764d24768e" exitCode=0 Nov 01 01:05:12.846540 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.846420 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" event={"ID":"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f","Type":"ContainerDied","Data":"0826badebb08010ed764eaf2b1e3d239013ab08822e6537b9dc0bb764d24768e"} Nov 01 01:05:12.855059 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.855036 6036 scope.go:122] "RemoveContainer" containerID="2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532" Nov 01 01:05:12.855436 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:05:12.855409 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532\": container with ID starting with 2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532 not found: ID does not exist" containerID="2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532" Nov 01 01:05:12.855528 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.855446 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532"} err="failed to get container status \"2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532\": rpc error: code = NotFound desc = could not find container \"2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532\": container with ID starting with 2f825b41a529386e1e444a9003172dda20db05d1593a2e83b198a9bfc3c5e532 not found: ID does not exist" Nov 01 01:05:12.927744 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.927716 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:05:12.929914 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.929895 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-kube-api-access-mbjzk\" (UniqueName: \"kubernetes.io/projected/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-kube-api-access-mbjzk\") pod \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\" (UID: \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\") " Nov 01 01:05:12.929983 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.929926 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-plugins-config-volume\") pod \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\" (UID: \"f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1\") " Nov 01 01:05:12.930441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.930419 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-plugins-config-volume" pod "f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1" (UID: "f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:12.932098 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:12.932078 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-kube-api-access-mbjzk" pod "f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1" (UID: "f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1"). InnerVolumeSpecName "kube-api-access-mbjzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:13.030908 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.030877 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-kube-api-access-nmxxc\" (UniqueName: \"kubernetes.io/projected/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-kube-api-access-nmxxc\") pod \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\" (UID: \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\") " Nov 01 01:05:13.031181 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.031162 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-plugins-config-volume\") pod \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\" (UID: \"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f\") " Nov 01 01:05:13.031581 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.031554 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-plugins-config-volume" pod "712175d1-b5eb-449f-baaa-8ffdd8c7fe4f" (UID: "712175d1-b5eb-449f-baaa-8ffdd8c7fe4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:13.031856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.031839 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjzk\" (UniqueName: \"kubernetes.io/projected/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-kube-api-access-mbjzk\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:13.031917 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.031863 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:13.031917 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.031877 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:13.033180 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.033159 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-kube-api-access-nmxxc" pod "712175d1-b5eb-449f-baaa-8ffdd8c7fe4f" (UID: "712175d1-b5eb-449f-baaa-8ffdd8c7fe4f"). InnerVolumeSpecName "kube-api-access-nmxxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:13.133871 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.133792 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxxc\" (UniqueName: \"kubernetes.io/projected/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f-kube-api-access-nmxxc\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:13.268510 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.268480 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4"] Nov 01 01:05:13.274217 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.274192 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-de2a95-7599764f98-4q2f4"] Nov 01 01:05:13.854413 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.854324 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" Nov 01 01:05:13.854413 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.854335 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f" event={"ID":"712175d1-b5eb-449f-baaa-8ffdd8c7fe4f","Type":"ContainerDied","Data":"4e7c3a4b07d201672cb354806d08f5376eb5eaffa94e24ea7333a81926b3c739"} Nov 01 01:05:13.854413 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:13.854397 6036 scope.go:122] "RemoveContainer" containerID="0826badebb08010ed764eaf2b1e3d239013ab08822e6537b9dc0bb764d24768e" Nov 01 01:05:14.023869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:14.023737 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f"] Nov 01 01:05:14.074662 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:14.074629 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-de2a95-85b97d978c-2db5f"] Nov 01 01:05:15.251179 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:15.251146 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712175d1-b5eb-449f-baaa-8ffdd8c7fe4f" path="/var/lib/kubelet/pods/712175d1-b5eb-449f-baaa-8ffdd8c7fe4f/volumes" Nov 01 01:05:15.251597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:15.251575 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1" path="/var/lib/kubelet/pods/f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1/volumes" Nov 01 01:05:16.264544 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:16.264508 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:16.473982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:16.473948 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:16.809559 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:16.809525 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:17.031790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:17.031574 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:19.506557 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:05:19.506524 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:05:22.380937 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:22.380899 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:27.301583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:27.301551 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:27.517199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:27.517159 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:27.723707 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:27.723624 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:31.257747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:31.257712 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:31.468665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:31.468629 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:32.347756 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:32.347708 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:34.611831 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:34.611793 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:37.233988 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:37.233948 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:42.011295 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:42.011240 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:46.897564 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:46.897520 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:49.506368 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:05:49.506329 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:05:51.153118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:51.153084 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:52.740065 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:52.740024 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:53.030596 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:53.030488 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:53.285690 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:53.285659 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:53.744520 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:53.744435 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:58.161195 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:58.161154 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:59.758890 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:59.758841 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:05:59.985758 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:05:59.985717 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:00.192150 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:00.192112 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:02.967981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:02.967943 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:04.087796 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.087739 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc"] Nov 01 01:06:04.088809 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.088773 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" podUID="e00b5366-0556-4856-9441-ad512c669362" containerName="maas-api" containerID="cri-o://1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0" gracePeriod=30 Nov 01 01:06:04.214245 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.214210 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:04.520148 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.520116 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:06:04.583119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.581616 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00b5366-0556-4856-9441-ad512c669362-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00b5366-0556-4856-9441-ad512c669362-openshift-service-ca\") pod \"e00b5366-0556-4856-9441-ad512c669362\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " Nov 01 01:06:04.583119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.581726 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-metrics-certs\") pod \"e00b5366-0556-4856-9441-ad512c669362\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " Nov 01 01:06:04.583119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.581784 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00b5366-0556-4856-9441-ad512c669362-kube-api-access-spb4z\" (UniqueName: \"kubernetes.io/projected/e00b5366-0556-4856-9441-ad512c669362-kube-api-access-spb4z\") pod \"e00b5366-0556-4856-9441-ad512c669362\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " Nov 01 01:06:04.583119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.581942 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls\") pod \"e00b5366-0556-4856-9441-ad512c669362\" (UID: \"e00b5366-0556-4856-9441-ad512c669362\") " Nov 01 01:06:04.586905 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.584266 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00b5366-0556-4856-9441-ad512c669362-openshift-service-ca" pod "e00b5366-0556-4856-9441-ad512c669362" (UID: "e00b5366-0556-4856-9441-ad512c669362"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:04.589234 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.588469 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00b5366-0556-4856-9441-ad512c669362-kube-api-access-spb4z" pod "e00b5366-0556-4856-9441-ad512c669362" (UID: "e00b5366-0556-4856-9441-ad512c669362"). InnerVolumeSpecName "kube-api-access-spb4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:04.590521 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.590496 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-metrics-certs" pod "e00b5366-0556-4856-9441-ad512c669362" (UID: "e00b5366-0556-4856-9441-ad512c669362"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:06:04.591156 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.591133 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls" pod "e00b5366-0556-4856-9441-ad512c669362" (UID: "e00b5366-0556-4856-9441-ad512c669362"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:06:04.682208 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.682047 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:04.683640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.683533 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:04.683640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.683557 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00b5366-0556-4856-9441-ad512c669362-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:04.683640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.683579 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00b5366-0556-4856-9441-ad512c669362-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:04.683640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:04.683588 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spb4z\" (UniqueName: \"kubernetes.io/projected/e00b5366-0556-4856-9441-ad512c669362-kube-api-access-spb4z\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.043936 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.043895 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:05.225269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.225235 6036 generic.go:356] "Generic (PLEG): container finished" podID="e00b5366-0556-4856-9441-ad512c669362" containerID="1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0" exitCode=0 Nov 01 01:06:05.225989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.225954 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" event={"ID":"e00b5366-0556-4856-9441-ad512c669362","Type":"ContainerDied","Data":"1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0"} Nov 01 01:06:05.226133 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.226026 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" event={"ID":"e00b5366-0556-4856-9441-ad512c669362","Type":"ContainerDied","Data":"bc6fdfb6542321db1306c7ed3b4421cf05f95887ee9250c2a05841867cfe140d"} Nov 01 01:06:05.226133 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.226054 6036 scope.go:122] "RemoveContainer" containerID="1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0" Nov 01 01:06:05.226253 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.226205 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc" Nov 01 01:06:05.300683 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.300664 6036 scope.go:122] "RemoveContainer" containerID="1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0" Nov 01 01:06:05.300975 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:06:05.300954 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0\": container with ID starting with 1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0 not found: ID does not exist" containerID="1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0" Nov 01 01:06:05.301015 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.300986 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0"} err="failed to get container status \"1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0\": rpc error: code = NotFound desc = could not find container \"1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0\": container with ID starting with 1da1094e35243c3534fa95e2eff2d07463926a92d724b04fd38fe84c1d5d02e0 not found: ID does not exist" Nov 01 01:06:05.417349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.417318 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc"] Nov 01 01:06:05.473075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.473028 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-de2a95-b766875f9-45bqc"] Nov 01 01:06:05.897599 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:05.897570 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:06.120639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:06.120607 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:06.468312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:06.456609 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:06.670340 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:06.667792 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:07.071919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:07.071874 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:07.277260 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:07.277224 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00b5366-0556-4856-9441-ad512c669362" path="/var/lib/kubelet/pods/e00b5366-0556-4856-9441-ad512c669362/volumes" Nov 01 01:06:11.743324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:11.743237 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:12.066986 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.066946 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:12.295988 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.295951 6036 generic.go:356] "Generic (PLEG): container finished" podID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerID="807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181" exitCode=0 Nov 01 01:06:12.296254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.296025 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" event={"ID":"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6","Type":"ContainerDied","Data":"807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181"} Nov 01 01:06:12.296616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.296594 6036 scope.go:122] "RemoveContainer" containerID="807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181" Nov 01 01:06:12.298468 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.298436 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:12.298550 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.298510 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:12.298550 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.298543 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:12.331250 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.331219 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:12.540668 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.540624 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp"] Nov 01 01:06:12.579264 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.579183 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:12.633147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.633110 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh"] Nov 01 01:06:12.974256 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:12.973935 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:13.197021 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:13.196980 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:13.305659 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:13.305628 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" event={"ID":"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6","Type":"ContainerStarted","Data":"d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe"} Nov 01 01:06:13.440256 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:13.440220 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:06:13.440500 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:13.440476 6036 patch_prober.go:29] interesting pod/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 01:06:13.440562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:13.440540 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 01:06:14.208168 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:14.208136 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:14.439933 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:14.439903 6036 patch_prober.go:29] interesting pod/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 01:06:14.440128 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:14.439958 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 01:06:14.465567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:14.465396 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:14.684105 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:14.684014 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:15.435359 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:15.435302 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:15.440573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:15.440548 6036 patch_prober.go:29] interesting pod/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 01:06:15.440671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:15.440604 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 01:06:16.093543 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:16.093494 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:16.440091 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:16.439998 6036 patch_prober.go:29] interesting pod/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 01:06:16.440091 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:16.440071 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 01:06:16.441041 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:16.441019 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:17.145110 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:17.145071 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:17.440339 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:17.440197 6036 patch_prober.go:29] interesting pod/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 01:06:17.440747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:17.440356 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 01:06:17.442819 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:17.442792 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:17.551744 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:17.551701 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" podUID="664423ef-4053-4db9-b4a4-85a93fb2e4fd" containerName="payload-processing" containerID="cri-o://4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e" gracePeriod=25 Nov 01 01:06:17.634381 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:17.634334 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" podUID="85dd805a-c7b1-49fb-9767-bc0309882f04" containerName="payload-pre-processing" containerID="cri-o://3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec" gracePeriod=25 Nov 01 01:06:17.669498 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:17.668826 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:18.022325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.022217 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 01:06:18.076963 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.076921 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664423ef-4053-4db9-b4a4-85a93fb2e4fd-kube-api-access-gsw6l\" (UniqueName: \"kubernetes.io/projected/664423ef-4053-4db9-b4a4-85a93fb2e4fd-kube-api-access-gsw6l\") pod \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\" (UID: \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\") " Nov 01 01:06:18.077143 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.077061 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664423ef-4053-4db9-b4a4-85a93fb2e4fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664423ef-4053-4db9-b4a4-85a93fb2e4fd-plugins-config-volume\") pod \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\" (UID: \"664423ef-4053-4db9-b4a4-85a93fb2e4fd\") " Nov 01 01:06:18.077463 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.077436 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664423ef-4053-4db9-b4a4-85a93fb2e4fd-plugins-config-volume" pod "664423ef-4053-4db9-b4a4-85a93fb2e4fd" (UID: "664423ef-4053-4db9-b4a4-85a93fb2e4fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:18.077780 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.077724 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664423ef-4053-4db9-b4a4-85a93fb2e4fd-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:18.079988 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.079962 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664423ef-4053-4db9-b4a4-85a93fb2e4fd-kube-api-access-gsw6l" pod "664423ef-4053-4db9-b4a4-85a93fb2e4fd" (UID: "664423ef-4053-4db9-b4a4-85a93fb2e4fd"). InnerVolumeSpecName "kube-api-access-gsw6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:18.082323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.082302 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 01:06:18.179092 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.179045 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85dd805a-c7b1-49fb-9767-bc0309882f04-kube-api-access-75zcm\" (UniqueName: \"kubernetes.io/projected/85dd805a-c7b1-49fb-9767-bc0309882f04-kube-api-access-75zcm\") pod \"85dd805a-c7b1-49fb-9767-bc0309882f04\" (UID: \"85dd805a-c7b1-49fb-9767-bc0309882f04\") " Nov 01 01:06:18.179346 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.179197 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85dd805a-c7b1-49fb-9767-bc0309882f04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dd805a-c7b1-49fb-9767-bc0309882f04-plugins-config-volume\") pod \"85dd805a-c7b1-49fb-9767-bc0309882f04\" (UID: \"85dd805a-c7b1-49fb-9767-bc0309882f04\") " Nov 01 01:06:18.179709 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.179679 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85dd805a-c7b1-49fb-9767-bc0309882f04-plugins-config-volume" pod "85dd805a-c7b1-49fb-9767-bc0309882f04" (UID: "85dd805a-c7b1-49fb-9767-bc0309882f04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:18.179779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.179697 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsw6l\" (UniqueName: \"kubernetes.io/projected/664423ef-4053-4db9-b4a4-85a93fb2e4fd-kube-api-access-gsw6l\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:18.181495 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.181469 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dd805a-c7b1-49fb-9767-bc0309882f04-kube-api-access-75zcm" pod "85dd805a-c7b1-49fb-9767-bc0309882f04" (UID: "85dd805a-c7b1-49fb-9767-bc0309882f04"). InnerVolumeSpecName "kube-api-access-75zcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:18.282448 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.282407 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zcm\" (UniqueName: \"kubernetes.io/projected/85dd805a-c7b1-49fb-9767-bc0309882f04-kube-api-access-75zcm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:18.282448 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.282435 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dd805a-c7b1-49fb-9767-bc0309882f04-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:18.340829 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.340799 6036 generic.go:356] "Generic (PLEG): container finished" podID="85dd805a-c7b1-49fb-9767-bc0309882f04" containerID="3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec" exitCode=0 Nov 01 01:06:18.341002 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.340844 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" Nov 01 01:06:18.341002 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.340890 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" event={"ID":"85dd805a-c7b1-49fb-9767-bc0309882f04","Type":"ContainerDied","Data":"3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec"} Nov 01 01:06:18.341002 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.340944 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh" event={"ID":"85dd805a-c7b1-49fb-9767-bc0309882f04","Type":"ContainerDied","Data":"144a5a1f51584da619af522524cab0da85c078dbc9513b76de0748216c2ca437"} Nov 01 01:06:18.341002 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.340978 6036 scope.go:122] "RemoveContainer" containerID="3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec" Nov 01 01:06:18.342427 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.342403 6036 generic.go:356] "Generic (PLEG): container finished" podID="664423ef-4053-4db9-b4a4-85a93fb2e4fd" containerID="4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e" exitCode=0 Nov 01 01:06:18.342561 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.342453 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" Nov 01 01:06:18.342561 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.342492 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" event={"ID":"664423ef-4053-4db9-b4a4-85a93fb2e4fd","Type":"ContainerDied","Data":"4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e"} Nov 01 01:06:18.342561 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.342541 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp" event={"ID":"664423ef-4053-4db9-b4a4-85a93fb2e4fd","Type":"ContainerDied","Data":"7b6a41781a7e584a8a5cd044d42a7d4222012111bb716a7c1916da6b49a91b15"} Nov 01 01:06:18.378794 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.378774 6036 scope.go:122] "RemoveContainer" containerID="3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec" Nov 01 01:06:18.379167 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:06:18.379148 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec\": container with ID starting with 3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec not found: ID does not exist" containerID="3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec" Nov 01 01:06:18.379215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.379177 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec"} err="failed to get container status \"3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec\": rpc error: code = NotFound desc = could not find container \"3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec\": container with ID starting with 3ddd65d93f7d2a00141fa8b16e78812abc1b79231a41bbd2a3f8717b7af086ec not found: ID does not exist" Nov 01 01:06:18.379215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.379195 6036 scope.go:122] "RemoveContainer" containerID="4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e" Nov 01 01:06:18.389261 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.389240 6036 scope.go:122] "RemoveContainer" containerID="4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e" Nov 01 01:06:18.389620 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:06:18.389600 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e\": container with ID starting with 4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e not found: ID does not exist" containerID="4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e" Nov 01 01:06:18.389671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.389629 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e"} err="failed to get container status \"4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e\": rpc error: code = NotFound desc = could not find container \"4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e\": container with ID starting with 4893978df954b2c04a8e14029748141af946a0e202041354e1de28eb30e1d76e not found: ID does not exist" Nov 01 01:06:18.439766 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.439731 6036 patch_prober.go:29] interesting pod/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 01:06:18.439942 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.439792 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 01:06:18.518110 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.518076 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh"] Nov 01 01:06:18.522708 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.522681 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-44c9bf-64f698c559-xc9lh"] Nov 01 01:06:18.636045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.636001 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp"] Nov 01 01:06:18.638816 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:18.638780 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-44c9bf-854c599b76-868xp"] Nov 01 01:06:19.252199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:19.252160 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664423ef-4053-4db9-b4a4-85a93fb2e4fd" path="/var/lib/kubelet/pods/664423ef-4053-4db9-b4a4-85a93fb2e4fd/volumes" Nov 01 01:06:19.252649 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:19.252635 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85dd805a-c7b1-49fb-9767-bc0309882f04" path="/var/lib/kubelet/pods/85dd805a-c7b1-49fb-9767-bc0309882f04/volumes" Nov 01 01:06:19.252940 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:19.252928 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:06:19.279773 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:19.279742 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv"] Nov 01 01:06:19.444748 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:19.444719 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:06:19.445869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:19.445846 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:06:19.506644 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:19.506560 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:19.507419 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:06:19.507394 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:06:19.877826 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:19.877797 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:20.132231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:20.132143 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:20.393027 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:20.392930 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:22.833463 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:22.833433 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:23.231146 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.231069 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:23.510121 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.510031 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:23.524759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.524731 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd"] Nov 01 01:06:23.525364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525351 6036 cpu_manager.go:414] "containerMap: removing container" podUID="e00b5366-0556-4856-9441-ad512c669362" containerName="maas-api" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525365 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00b5366-0556-4856-9441-ad512c669362" containerName="maas-api" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525378 6036 cpu_manager.go:414] "containerMap: removing container" podUID="394c5e6a-ff94-4550-b421-7686fa50097b" containerName="payload-pre-processing" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525383 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394c5e6a-ff94-4550-b421-7686fa50097b" containerName="payload-pre-processing" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525402 6036 cpu_manager.go:414] "containerMap: removing container" podUID="f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1" containerName="payload-processing" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525406 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1" containerName="payload-processing" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525419 6036 cpu_manager.go:414] "containerMap: removing container" podUID="664423ef-4053-4db9-b4a4-85a93fb2e4fd" containerName="payload-processing" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525424 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664423ef-4053-4db9-b4a4-85a93fb2e4fd" containerName="payload-processing" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525430 6036 cpu_manager.go:414] "containerMap: removing container" podUID="712175d1-b5eb-449f-baaa-8ffdd8c7fe4f" containerName="payload-pre-processing" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525434 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712175d1-b5eb-449f-baaa-8ffdd8c7fe4f" containerName="payload-pre-processing" Nov 01 01:06:23.525443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525446 6036 cpu_manager.go:414] "containerMap: removing container" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525451 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525467 6036 cpu_manager.go:414] "containerMap: removing container" podUID="91185e42-540e-4fd4-bf03-f52783251c1f" containerName="payload-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525471 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91185e42-540e-4fd4-bf03-f52783251c1f" containerName="payload-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525505 6036 cpu_manager.go:414] "containerMap: removing container" podUID="85dd805a-c7b1-49fb-9767-bc0309882f04" containerName="payload-pre-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525511 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85dd805a-c7b1-49fb-9767-bc0309882f04" containerName="payload-pre-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525604 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525613 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00b5366-0556-4856-9441-ad512c669362" containerName="maas-api" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525623 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="394c5e6a-ff94-4550-b421-7686fa50097b" containerName="payload-pre-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525630 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fb71c9-8b5e-4b46-bcdb-b875d3f0c2b1" containerName="payload-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525637 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="664423ef-4053-4db9-b4a4-85a93fb2e4fd" containerName="payload-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525644 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="712175d1-b5eb-449f-baaa-8ffdd8c7fe4f" containerName="payload-pre-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525653 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="91185e42-540e-4fd4-bf03-f52783251c1f" containerName="payload-processing" Nov 01 01:06:23.525821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.525661 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="85dd805a-c7b1-49fb-9767-bc0309882f04" containerName="payload-pre-processing" Nov 01 01:06:23.530748 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.530730 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.586836 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.586808 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b112c158-openshift-default-dockercfg-jnjjd\"" Nov 01 01:06:23.640849 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.640814 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd"] Nov 01 01:06:23.670856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.670818 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.671078 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.670938 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.671078 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.671046 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b84sd\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-kube-api-access-b84sd\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.671212 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.671195 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.671271 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.671239 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d426904-92e2-427c-b318-2a841fec00e7-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.671401 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.671381 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.671478 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.671422 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d426904-92e2-427c-b318-2a841fec00e7-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.671537 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.671507 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.671595 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.671575 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786300 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786245 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786500 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786376 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786500 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786420 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84sd\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-kube-api-access-b84sd\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786508 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786547 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d426904-92e2-427c-b318-2a841fec00e7-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786606 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786844 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786637 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d426904-92e2-427c-b318-2a841fec00e7-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786844 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786714 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.786844 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.786767 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.794981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.794950 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.796765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.795924 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.796946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.795972 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.797043 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.796174 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.797132 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.796703 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d426904-92e2-427c-b318-2a841fec00e7-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.803656 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.803467 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.804390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.804364 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d426904-92e2-427c-b318-2a841fec00e7-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.858746 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.858712 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b84sd\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-kube-api-access-b84sd\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:23.865033 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:23.864999 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:24.144363 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:24.144234 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:24.580027 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:24.580003 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd"] Nov 01 01:06:24.580860 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:06:24.580821 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d426904_92e2_427c_b318_2a841fec00e7.slice/crio-a2b4f5c081605967688fd7b47c80ed444fe937cb2ea8286ceacd2f130b7cb299 WatchSource:0}: Error finding container a2b4f5c081605967688fd7b47c80ed444fe937cb2ea8286ceacd2f130b7cb299: Status 404 returned error can't find the container with id a2b4f5c081605967688fd7b47c80ed444fe937cb2ea8286ceacd2f130b7cb299 Nov 01 01:06:24.584007 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:24.583966 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:24.584091 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:24.584058 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:24.584127 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:24.584098 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:25.400614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:25.400571 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" event={"ID":"3d426904-92e2-427c-b318-2a841fec00e7","Type":"ContainerStarted","Data":"1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c"} Nov 01 01:06:25.401238 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:25.401211 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" event={"ID":"3d426904-92e2-427c-b318-2a841fec00e7","Type":"ContainerStarted","Data":"a2b4f5c081605967688fd7b47c80ed444fe937cb2ea8286ceacd2f130b7cb299"} Nov 01 01:06:25.595270 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:25.595202 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podStartSLOduration=2.595182882 podStartE2EDuration="2.595182882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:25.592889959 +0000 UTC m=+2530.468110973" watchObservedRunningTime="2026-11-01 01:06:25.595182882 +0000 UTC m=+2530.470403899" Nov 01 01:06:26.145064 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:26.145029 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:26.167312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:26.164749 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 01:06:26.167312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:26.164825 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 01:06:27.119308 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:27.119134 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:27.144830 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:27.144799 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 01:06:27.145001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:27.144857 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 01:06:27.371879 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:27.371774 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:28.024223 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:28.024074 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:28.148335 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:28.146702 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 01:06:28.148335 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:28.146761 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 01:06:28.720786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:28.720752 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:28.944108 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:28.944077 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:29.144622 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:29.144589 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 01:06:29.144783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:29.144656 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 01:06:30.152317 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:30.145613 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 01:06:30.152317 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:30.145671 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 01:06:31.145215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:31.145175 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 01:06:31.145427 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:31.145247 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 01:06:32.150652 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:32.150619 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:32.151142 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:32.150973 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:32.151856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:32.151838 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:06:32.413382 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:32.413203 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:34.302968 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.302926 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" containerID="cri-o://d781f0ed6c8841f30ebfe496ed7966e720c91d7030b04e04e859160f1a1351a5" gracePeriod=45 Nov 01 01:06:34.345973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.345938 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:34.464306 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.464255 6036 generic.go:356] "Generic (PLEG): container finished" podID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerID="d781f0ed6c8841f30ebfe496ed7966e720c91d7030b04e04e859160f1a1351a5" exitCode=0 Nov 01 01:06:34.464501 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.464332 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" event={"ID":"bec0460f-fcb9-48c2-a465-a9a0845177e6","Type":"ContainerDied","Data":"d781f0ed6c8841f30ebfe496ed7966e720c91d7030b04e04e859160f1a1351a5"} Nov 01 01:06:34.555789 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.555711 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:34.575526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.575499 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 01:06:34.677531 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677496 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-home\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-home\") pod \"bec0460f-fcb9-48c2-a465-a9a0845177e6\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " Nov 01 01:06:34.677692 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677555 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-model-cache\") pod \"bec0460f-fcb9-48c2-a465-a9a0845177e6\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " Nov 01 01:06:34.677692 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677580 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec0460f-fcb9-48c2-a465-a9a0845177e6-kube-api-access-knrtr\" (UniqueName: \"kubernetes.io/projected/bec0460f-fcb9-48c2-a465-a9a0845177e6-kube-api-access-knrtr\") pod \"bec0460f-fcb9-48c2-a465-a9a0845177e6\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " Nov 01 01:06:34.677692 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677610 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-tmp-dir\") pod \"bec0460f-fcb9-48c2-a465-a9a0845177e6\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " Nov 01 01:06:34.677805 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677739 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-dshm\") pod \"bec0460f-fcb9-48c2-a465-a9a0845177e6\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " Nov 01 01:06:34.677805 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677781 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-home" pod "bec0460f-fcb9-48c2-a465-a9a0845177e6" (UID: "bec0460f-fcb9-48c2-a465-a9a0845177e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:34.677907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677861 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-model-cache" pod "bec0460f-fcb9-48c2-a465-a9a0845177e6" (UID: "bec0460f-fcb9-48c2-a465-a9a0845177e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:34.677948 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677910 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-tmp-dir" pod "bec0460f-fcb9-48c2-a465-a9a0845177e6" (UID: "bec0460f-fcb9-48c2-a465-a9a0845177e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:34.677948 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.677870 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec0460f-fcb9-48c2-a465-a9a0845177e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0460f-fcb9-48c2-a465-a9a0845177e6-tls-certs\") pod \"bec0460f-fcb9-48c2-a465-a9a0845177e6\" (UID: \"bec0460f-fcb9-48c2-a465-a9a0845177e6\") " Nov 01 01:06:34.678661 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.678637 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:34.678661 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.678660 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:34.678863 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.678671 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:34.680100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.680077 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec0460f-fcb9-48c2-a465-a9a0845177e6-kube-api-access-knrtr" pod "bec0460f-fcb9-48c2-a465-a9a0845177e6" (UID: "bec0460f-fcb9-48c2-a465-a9a0845177e6"). InnerVolumeSpecName "kube-api-access-knrtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:34.680100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.680088 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec0460f-fcb9-48c2-a465-a9a0845177e6-tls-certs" pod "bec0460f-fcb9-48c2-a465-a9a0845177e6" (UID: "bec0460f-fcb9-48c2-a465-a9a0845177e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:06:34.680100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.680095 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-dshm" pod "bec0460f-fcb9-48c2-a465-a9a0845177e6" (UID: "bec0460f-fcb9-48c2-a465-a9a0845177e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:34.780466 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.780433 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0460f-fcb9-48c2-a465-a9a0845177e6-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:34.780620 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.780470 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knrtr\" (UniqueName: \"kubernetes.io/projected/bec0460f-fcb9-48c2-a465-a9a0845177e6-kube-api-access-knrtr\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:34.780620 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.780488 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0460f-fcb9-48c2-a465-a9a0845177e6-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:34.792389 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:34.792356 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:35.022902 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:35.022803 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:35.443771 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:35.443730 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:35.473725 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:35.473698 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" Nov 01 01:06:35.473725 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:35.473712 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv" event={"ID":"bec0460f-fcb9-48c2-a465-a9a0845177e6","Type":"ContainerDied","Data":"bddb25c40b2319930d240ab3d59e987b32ce400484af7c0b5c7c49d38f0fada0"} Nov 01 01:06:35.473954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:35.473763 6036 scope.go:122] "RemoveContainer" containerID="d781f0ed6c8841f30ebfe496ed7966e720c91d7030b04e04e859160f1a1351a5" Nov 01 01:06:35.642980 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:35.642945 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv"] Nov 01 01:06:35.694498 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:35.694393 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4143b7/premium-w2-4143b7-kserve-f4c88f8f9-mnsrv"] Nov 01 01:06:37.252279 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:37.252243 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" path="/var/lib/kubelet/pods/bec0460f-fcb9-48c2-a465-a9a0845177e6/volumes" Nov 01 01:06:38.644688 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.644657 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4"] Nov 01 01:06:38.645351 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.645334 6036 cpu_manager.go:414] "containerMap: removing container" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" Nov 01 01:06:38.645442 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.645351 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" Nov 01 01:06:38.645442 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.645407 6036 cpu_manager.go:414] "containerMap: removing container" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" Nov 01 01:06:38.645442 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.645413 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" Nov 01 01:06:38.645539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.645514 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f402208-c342-4429-8b0b-3259e410455d" containerName="istio-proxy" Nov 01 01:06:38.645539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.645526 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec0460f-fcb9-48c2-a465-a9a0845177e6" containerName="main" Nov 01 01:06:38.650244 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.650226 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.695947 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.695911 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g"] Nov 01 01:06:38.700352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.700331 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.737714 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.737675 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bx2\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-kube-api-access-m6bx2\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.737714 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.737716 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.737956 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.737746 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjktw\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-kube-api-access-wjktw\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.737956 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.737833 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.737956 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.737881 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.737956 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.737899 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.738177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.737970 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.738177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738096 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.738177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738166 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.738377 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738237 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.738377 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738348 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.738490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738374 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.738550 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738484 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.738608 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738548 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.738608 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738596 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.738722 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738659 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.738722 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738686 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.738833 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.738797 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.754528 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.754499 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4"] Nov 01 01:06:38.806703 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.806667 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g"] Nov 01 01:06:38.840538 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840501 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.840737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840555 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.840737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840605 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.840737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840647 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.840737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840728 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.840981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840752 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.840981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840813 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.840981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840843 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.840981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840890 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.840981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840916 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.840981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.840941 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.841344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841001 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.841344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841049 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bx2\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-kube-api-access-m6bx2\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.841344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841053 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.841344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841077 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.841344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841125 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjktw\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-kube-api-access-wjktw\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.841344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841160 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.841344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841228 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.841344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841256 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.842005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841424 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.842005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841687 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.842005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841856 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.842005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841955 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.842005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.841978 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.842324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.842119 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.842324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.842191 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.842324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.842215 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.842494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.842391 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.843721 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.843693 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.844926 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.844895 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.845060 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.845012 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.845157 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.845135 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.905915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.905825 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bx2\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-kube-api-access-m6bx2\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.906349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.906320 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:38.906654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.906634 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.915397 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.915367 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjktw\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-kube-api-access-wjktw\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:38.963865 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:38.963829 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:39.022939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.022586 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:39.399088 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.399048 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4"] Nov 01 01:06:39.399579 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:06:39.399556 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c7cb30_f378_47ee_8b85_3a431de3b7e8.slice/crio-6a318b281a1e83a07fe9a958e5fd7e3f7225c36ca9d6f5b09263ea0baf297598 WatchSource:0}: Error finding container 6a318b281a1e83a07fe9a958e5fd7e3f7225c36ca9d6f5b09263ea0baf297598: Status 404 returned error can't find the container with id 6a318b281a1e83a07fe9a958e5fd7e3f7225c36ca9d6f5b09263ea0baf297598 Nov 01 01:06:39.402476 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.402439 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:39.402584 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.402527 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:39.402584 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.402572 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:39.489118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.489082 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:39.492625 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.492579 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:39.492803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.492723 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:39.492803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.492766 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g"] Nov 01 01:06:39.492956 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.492815 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:39.512721 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.512690 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" event={"ID":"41c7cb30-f378-47ee-8b85-3a431de3b7e8","Type":"ContainerStarted","Data":"6a318b281a1e83a07fe9a958e5fd7e3f7225c36ca9d6f5b09263ea0baf297598"} Nov 01 01:06:39.514272 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:39.514223 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" event={"ID":"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61","Type":"ContainerStarted","Data":"969434da28ab519cba759d1239bcb0df2f0c86345c41e41ed06dd3f165e054fe"} Nov 01 01:06:40.522850 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:40.522807 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" event={"ID":"41c7cb30-f378-47ee-8b85-3a431de3b7e8","Type":"ContainerStarted","Data":"b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af"} Nov 01 01:06:40.524507 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:40.524477 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" event={"ID":"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61","Type":"ContainerStarted","Data":"a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93"} Nov 01 01:06:40.812946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:40.812894 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podStartSLOduration=2.812878428 podStartE2EDuration="2.812878428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:40.810491057 +0000 UTC m=+2545.685712086" watchObservedRunningTime="2026-11-01 01:06:40.812878428 +0000 UTC m=+2545.688099442" Nov 01 01:06:40.813238 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:40.813219 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podStartSLOduration=2.813212133 podStartE2EDuration="2.813212133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:40.681689071 +0000 UTC m=+2545.556910087" watchObservedRunningTime="2026-11-01 01:06:40.813212133 +0000 UTC m=+2545.688433148" Nov 01 01:06:40.968391 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:40.968353 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:40.973316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:40.969576 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 01:06:40.973316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:40.969633 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 01:06:41.023760 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:41.023734 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:41.024768 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:41.024746 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 01:06:41.024889 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:41.024799 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 01:06:41.703318 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:41.695386 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:41.975365 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:41.974832 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:42.000313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:41.994378 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 01:06:42.000313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:41.994542 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 01:06:42.023274 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:42.023233 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 01:06:42.023484 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:42.023311 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 01:06:42.205149 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:42.204989 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:42.977313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:42.974000 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 01:06:42.977313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:42.974059 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 01:06:43.024224 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:43.024189 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 01:06:43.024474 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:43.024241 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 01:06:43.792316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:43.789739 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:43.966189 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:43.966155 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 01:06:43.966515 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:43.966484 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 01:06:44.002911 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:44.002875 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:44.024161 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:44.024127 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 01:06:44.024514 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:44.024481 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 01:06:44.654253 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:44.654213 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:44.974838 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:44.974737 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 01:06:44.975123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:44.975091 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 01:06:45.023507 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:45.023472 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 01:06:45.023901 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:45.023537 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 01:06:45.964951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:45.964907 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 01:06:45.965135 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:45.964993 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 01:06:46.023387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:46.023349 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 01:06:46.023568 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:46.023415 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 01:06:46.968615 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:46.968581 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:46.969075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:46.969062 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:46.969786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:46.969768 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:06:47.028024 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:47.027992 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:47.028422 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:47.028269 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:47.028945 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:47.028930 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:06:49.506921 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:06:49.506884 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:06:49.908929 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:49.908896 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:50.417857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:50.417822 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:51.168580 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:51.168535 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:51.841007 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:51.840971 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:52.413047 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:52.413008 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:54.327749 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:54.327714 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj"] Nov 01 01:06:54.433913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:54.432156 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj"] Nov 01 01:06:54.769843 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:54.769706 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:54.980443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:54.980412 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:56.604816 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:56.604781 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:56.812976 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:56.812938 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:56.942842 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:56.942750 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf"] Nov 01 01:06:57.027775 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:57.027744 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:57.053715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:57.053684 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f"] Nov 01 01:06:57.403691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:57.403651 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:57.612498 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:57.612467 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:57.868207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:57.868173 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:58.091777 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:58.091748 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:58.420700 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:58.420667 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:06:59.328465 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.328420 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" podUID="f2fb595d-d9fe-40e8-9f09-8eea308bf3d7" containerName="payload-processing" containerID="cri-o://894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6" gracePeriod=25 Nov 01 01:06:59.432644 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.432575 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" podUID="050fffbe-b376-4cbc-ab7f-8b1c76349986" containerName="payload-pre-processing" containerID="cri-o://6bc22843b42e5c555a501c241ecff82193bebd59510cb87ae7bc86be3b3acca5" gracePeriod=25 Nov 01 01:06:59.610347 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.610323 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.704726 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-kube-api-access-q8ddk\" (UniqueName: \"kubernetes.io/projected/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-kube-api-access-q8ddk\") pod \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\" (UID: \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\") " Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.704785 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-plugins-config-volume\") pod \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\" (UID: \"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7\") " Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.705615 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-plugins-config-volume" pod "f2fb595d-d9fe-40e8-9f09-8eea308bf3d7" (UID: "f2fb595d-d9fe-40e8-9f09-8eea308bf3d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.706563 6036 generic.go:356] "Generic (PLEG): container finished" podID="f2fb595d-d9fe-40e8-9f09-8eea308bf3d7" containerID="894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6" exitCode=0 Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.706683 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" event={"ID":"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7","Type":"ContainerDied","Data":"894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6"} Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.706725 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" event={"ID":"f2fb595d-d9fe-40e8-9f09-8eea308bf3d7","Type":"ContainerDied","Data":"364bfdda3e10ad27cb4e0f0a7e25f2b727878787d1b2a41dfd330792d8c679b0"} Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.706747 6036 scope.go:122] "RemoveContainer" containerID="894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6" Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.706856 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj" Nov 01 01:06:59.708562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.707219 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-kube-api-access-q8ddk" pod "f2fb595d-d9fe-40e8-9f09-8eea308bf3d7" (UID: "f2fb595d-d9fe-40e8-9f09-8eea308bf3d7"). InnerVolumeSpecName "kube-api-access-q8ddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:59.709259 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.709230 6036 generic.go:356] "Generic (PLEG): container finished" podID="050fffbe-b376-4cbc-ab7f-8b1c76349986" containerID="6bc22843b42e5c555a501c241ecff82193bebd59510cb87ae7bc86be3b3acca5" exitCode=0 Nov 01 01:06:59.709454 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.709261 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" event={"ID":"050fffbe-b376-4cbc-ab7f-8b1c76349986","Type":"ContainerDied","Data":"6bc22843b42e5c555a501c241ecff82193bebd59510cb87ae7bc86be3b3acca5"} Nov 01 01:06:59.729329 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.726159 6036 scope.go:122] "RemoveContainer" containerID="894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6" Nov 01 01:06:59.730170 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:06:59.730140 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6\": container with ID starting with 894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6 not found: ID does not exist" containerID="894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6" Nov 01 01:06:59.730396 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.730179 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6"} err="failed to get container status \"894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6\": rpc error: code = NotFound desc = could not find container \"894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6\": container with ID starting with 894f57af409bd3fa0d435ba1a554ef96b11cba93f06386524fe38179497c6ca6 not found: ID does not exist" Nov 01 01:06:59.734340 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.731354 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:06:59.807124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.807085 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ddk\" (UniqueName: \"kubernetes.io/projected/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-kube-api-access-q8ddk\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:59.807124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.807117 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:59.908075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.907987 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050fffbe-b376-4cbc-ab7f-8b1c76349986-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050fffbe-b376-4cbc-ab7f-8b1c76349986-plugins-config-volume\") pod \"050fffbe-b376-4cbc-ab7f-8b1c76349986\" (UID: \"050fffbe-b376-4cbc-ab7f-8b1c76349986\") " Nov 01 01:06:59.908075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.908058 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050fffbe-b376-4cbc-ab7f-8b1c76349986-kube-api-access-7lwrp\" (UniqueName: \"kubernetes.io/projected/050fffbe-b376-4cbc-ab7f-8b1c76349986-kube-api-access-7lwrp\") pod \"050fffbe-b376-4cbc-ab7f-8b1c76349986\" (UID: \"050fffbe-b376-4cbc-ab7f-8b1c76349986\") " Nov 01 01:06:59.908406 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.908384 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050fffbe-b376-4cbc-ab7f-8b1c76349986-plugins-config-volume" pod "050fffbe-b376-4cbc-ab7f-8b1c76349986" (UID: "050fffbe-b376-4cbc-ab7f-8b1c76349986"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:59.908560 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.908538 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050fffbe-b376-4cbc-ab7f-8b1c76349986-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:59.910455 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:06:59.910432 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050fffbe-b376-4cbc-ab7f-8b1c76349986-kube-api-access-7lwrp" pod "050fffbe-b376-4cbc-ab7f-8b1c76349986" (UID: "050fffbe-b376-4cbc-ab7f-8b1c76349986"). InnerVolumeSpecName "kube-api-access-7lwrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:00.009518 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.009485 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lwrp\" (UniqueName: \"kubernetes.io/projected/050fffbe-b376-4cbc-ab7f-8b1c76349986-kube-api-access-7lwrp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:00.132898 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.132443 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj"] Nov 01 01:07:00.138227 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.138193 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b112c158-8576bf7b66-64nvj"] Nov 01 01:07:00.719595 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.719492 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" Nov 01 01:07:00.719595 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.719502 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj" event={"ID":"050fffbe-b376-4cbc-ab7f-8b1c76349986","Type":"ContainerDied","Data":"ccd359894fcfd30657efe5d90ae902602003b78f8d3f900df2dc17045b4912b8"} Nov 01 01:07:00.720130 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.719601 6036 scope.go:122] "RemoveContainer" containerID="6bc22843b42e5c555a501c241ecff82193bebd59510cb87ae7bc86be3b3acca5" Nov 01 01:07:00.891412 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.891366 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:00.892871 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.892846 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj"] Nov 01 01:07:00.895632 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:00.895609 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b112c158-7cd8f94699-n4nmj"] Nov 01 01:07:01.097928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:01.097890 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:01.251308 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:01.251263 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050fffbe-b376-4cbc-ab7f-8b1c76349986" path="/var/lib/kubelet/pods/050fffbe-b376-4cbc-ab7f-8b1c76349986/volumes" Nov 01 01:07:01.251829 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:01.251807 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fb595d-d9fe-40e8-9f09-8eea308bf3d7" path="/var/lib/kubelet/pods/f2fb595d-d9fe-40e8-9f09-8eea308bf3d7/volumes" Nov 01 01:07:01.943159 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:01.943118 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" podUID="9c47c543-f8cd-4a7c-811f-2f884e97a10b" containerName="payload-processing" containerID="cri-o://5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86" gracePeriod=25 Nov 01 01:07:02.054271 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.054225 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" podUID="092dea27-ada1-4838-ab3b-a38b4dbc4d97" containerName="payload-pre-processing" containerID="cri-o://09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800" gracePeriod=25 Nov 01 01:07:02.231965 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.231930 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:07:02.339152 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.339126 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:07:02.349664 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.349627 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c47c543-f8cd-4a7c-811f-2f884e97a10b-kube-api-access-vf768\" (UniqueName: \"kubernetes.io/projected/9c47c543-f8cd-4a7c-811f-2f884e97a10b-kube-api-access-vf768\") pod \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\" (UID: \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\") " Nov 01 01:07:02.349923 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.349894 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c47c543-f8cd-4a7c-811f-2f884e97a10b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47c543-f8cd-4a7c-811f-2f884e97a10b-plugins-config-volume\") pod \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\" (UID: \"9c47c543-f8cd-4a7c-811f-2f884e97a10b\") " Nov 01 01:07:02.351319 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.350675 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c47c543-f8cd-4a7c-811f-2f884e97a10b-plugins-config-volume" pod "9c47c543-f8cd-4a7c-811f-2f884e97a10b" (UID: "9c47c543-f8cd-4a7c-811f-2f884e97a10b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:02.352686 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.352661 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c47c543-f8cd-4a7c-811f-2f884e97a10b-kube-api-access-vf768" pod "9c47c543-f8cd-4a7c-811f-2f884e97a10b" (UID: "9c47c543-f8cd-4a7c-811f-2f884e97a10b"). InnerVolumeSpecName "kube-api-access-vf768". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:02.451813 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.451712 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092dea27-ada1-4838-ab3b-a38b4dbc4d97-kube-api-access-tqmns\" (UniqueName: \"kubernetes.io/projected/092dea27-ada1-4838-ab3b-a38b4dbc4d97-kube-api-access-tqmns\") pod \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\" (UID: \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\") " Nov 01 01:07:02.452235 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.452200 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092dea27-ada1-4838-ab3b-a38b4dbc4d97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092dea27-ada1-4838-ab3b-a38b4dbc4d97-plugins-config-volume\") pod \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\" (UID: \"092dea27-ada1-4838-ab3b-a38b4dbc4d97\") " Nov 01 01:07:02.453017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.452983 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf768\" (UniqueName: \"kubernetes.io/projected/9c47c543-f8cd-4a7c-811f-2f884e97a10b-kube-api-access-vf768\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:02.453017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.453015 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47c543-f8cd-4a7c-811f-2f884e97a10b-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:02.454001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.453774 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092dea27-ada1-4838-ab3b-a38b4dbc4d97-plugins-config-volume" pod "092dea27-ada1-4838-ab3b-a38b4dbc4d97" (UID: "092dea27-ada1-4838-ab3b-a38b4dbc4d97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:02.456783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.456250 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092dea27-ada1-4838-ab3b-a38b4dbc4d97-kube-api-access-tqmns" pod "092dea27-ada1-4838-ab3b-a38b4dbc4d97" (UID: "092dea27-ada1-4838-ab3b-a38b4dbc4d97"). InnerVolumeSpecName "kube-api-access-tqmns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:02.553990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.553961 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmns\" (UniqueName: \"kubernetes.io/projected/092dea27-ada1-4838-ab3b-a38b4dbc4d97-kube-api-access-tqmns\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:02.553990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.553988 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092dea27-ada1-4838-ab3b-a38b4dbc4d97-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:02.735791 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.735695 6036 generic.go:356] "Generic (PLEG): container finished" podID="092dea27-ada1-4838-ab3b-a38b4dbc4d97" containerID="09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800" exitCode=0 Nov 01 01:07:02.735791 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.735755 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" Nov 01 01:07:02.736054 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.735802 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" event={"ID":"092dea27-ada1-4838-ab3b-a38b4dbc4d97","Type":"ContainerDied","Data":"09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800"} Nov 01 01:07:02.736054 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.735893 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f" event={"ID":"092dea27-ada1-4838-ab3b-a38b4dbc4d97","Type":"ContainerDied","Data":"66c9ee410ad682a8c546691b1bfe13ea7f49ed7546b175c0f04ce612db0cba9a"} Nov 01 01:07:02.736054 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.735949 6036 scope.go:122] "RemoveContainer" containerID="09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800" Nov 01 01:07:02.737900 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.737875 6036 generic.go:356] "Generic (PLEG): container finished" podID="9c47c543-f8cd-4a7c-811f-2f884e97a10b" containerID="5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86" exitCode=0 Nov 01 01:07:02.738041 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.737935 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" event={"ID":"9c47c543-f8cd-4a7c-811f-2f884e97a10b","Type":"ContainerDied","Data":"5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86"} Nov 01 01:07:02.738041 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.737954 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" Nov 01 01:07:02.738041 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.737999 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf" event={"ID":"9c47c543-f8cd-4a7c-811f-2f884e97a10b","Type":"ContainerDied","Data":"58e82058689e90218f850783ece977c08e8e4d8d97b1cbbca294369bd878e6ac"} Nov 01 01:07:02.748647 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.748571 6036 scope.go:122] "RemoveContainer" containerID="09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800" Nov 01 01:07:02.748929 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:02.748906 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800\": container with ID starting with 09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800 not found: ID does not exist" containerID="09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800" Nov 01 01:07:02.749015 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.748935 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800"} err="failed to get container status \"09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800\": rpc error: code = NotFound desc = could not find container \"09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800\": container with ID starting with 09f8c88c53ebc7da8c54de213984b46c65ea61e330ab01ee0b23b7d706e86800 not found: ID does not exist" Nov 01 01:07:02.749015 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.748952 6036 scope.go:122] "RemoveContainer" containerID="5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86" Nov 01 01:07:02.758346 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.758328 6036 scope.go:122] "RemoveContainer" containerID="5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86" Nov 01 01:07:02.758613 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:02.758593 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86\": container with ID starting with 5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86 not found: ID does not exist" containerID="5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86" Nov 01 01:07:02.758663 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.758622 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86"} err="failed to get container status \"5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86\": rpc error: code = NotFound desc = could not find container \"5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86\": container with ID starting with 5d142425cbd59c256be79d2529d871594ac01bd4a5943419acec46e63b99fb86 not found: ID does not exist" Nov 01 01:07:02.912677 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.912638 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf"] Nov 01 01:07:02.963636 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:02.963602 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f73fb0-5f445458d4-rnxkf"] Nov 01 01:07:03.075036 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:03.075008 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f"] Nov 01 01:07:03.081764 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:03.081444 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f73fb0-676bc9b867-qrv9f"] Nov 01 01:07:03.251617 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:03.251572 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092dea27-ada1-4838-ab3b-a38b4dbc4d97" path="/var/lib/kubelet/pods/092dea27-ada1-4838-ab3b-a38b4dbc4d97/volumes" Nov 01 01:07:03.252066 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:03.252053 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c47c543-f8cd-4a7c-811f-2f884e97a10b" path="/var/lib/kubelet/pods/9c47c543-f8cd-4a7c-811f-2f884e97a10b/volumes" Nov 01 01:07:04.271974 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:04.271936 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:04.484240 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:04.484204 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:05.635984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:05.635942 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:07.345185 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:07.345143 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:07.644942 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:07.644859 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:07.859465 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:07.859429 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:08.298774 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:08.298742 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:09.171642 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:09.171602 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:09.946724 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:09.946690 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:10.153215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:10.153184 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:12.988808 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:12.988754 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:14.440833 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:14.440797 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:14.653647 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:14.653614 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:14.685793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:14.685762 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf"] Nov 01 01:07:14.686005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:14.685979 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" containerID="cri-o://56744e4edb65cc9e69c4feb326675cef32f08cc3d890faa600751d0b56023bc2" gracePeriod=30 Nov 01 01:07:14.864080 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:14.864037 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:15.234872 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:15.234764 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:18.460033 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:18.459994 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:19.505430 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:19.505389 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:07:19.885676 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:19.885538 6036 generic.go:356] "Generic (PLEG): container finished" podID="d8844a5a-492a-4534-a3af-07def8126146" containerID="56744e4edb65cc9e69c4feb326675cef32f08cc3d890faa600751d0b56023bc2" exitCode=0 Nov 01 01:07:19.885676 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:19.885609 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" event={"ID":"d8844a5a-492a-4534-a3af-07def8126146","Type":"ContainerDied","Data":"56744e4edb65cc9e69c4feb326675cef32f08cc3d890faa600751d0b56023bc2"} Nov 01 01:07:19.885883 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:19.885684 6036 scope.go:122] "RemoveContainer" containerID="d57fbafa75c687c35b4d1cf3e902bcc2ddc8189c75569512625a567fdcd317d8" Nov 01 01:07:19.959722 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:19.959691 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 01:07:20.063951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.063871 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-certs\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.063951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.063906 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-istio-token\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-istio-token\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.063951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.063929 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-credential-socket\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.064223 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.063981 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-kube-api-access-nsr9l\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-kube-api-access-nsr9l\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.064223 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.064106 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-data\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.064223 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.064169 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-certs" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:20.064223 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.064207 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8844a5a-492a-4534-a3af-07def8126146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8844a5a-492a-4534-a3af-07def8126146-istiod-ca-cert\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.064494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.064262 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-envoy\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.064494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.064443 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8844a5a-492a-4534-a3af-07def8126146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8844a5a-492a-4534-a3af-07def8126146-istio-podinfo\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.064494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.064459 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-data" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:20.064653 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.064505 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-credential-socket" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:20.064868 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.064845 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8844a5a-492a-4534-a3af-07def8126146-istiod-ca-cert" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:20.066434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.065078 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-socket\") pod \"d8844a5a-492a-4534-a3af-07def8126146\" (UID: \"d8844a5a-492a-4534-a3af-07def8126146\") " Nov 01 01:07:20.066434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.065350 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-socket" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:20.066434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.065811 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.066434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.065832 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.066434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.065848 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.066434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.065862 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.066434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.065876 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8844a5a-492a-4534-a3af-07def8126146-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.066801 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.066704 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-envoy" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:20.066911 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.066890 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-istio-token" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:20.067011 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.066995 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-kube-api-access-nsr9l" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "kube-api-access-nsr9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:20.067781 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.067759 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8844a5a-492a-4534-a3af-07def8126146-istio-podinfo" pod "d8844a5a-492a-4534-a3af-07def8126146" (UID: "d8844a5a-492a-4534-a3af-07def8126146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:07:20.125955 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.125919 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr"] Nov 01 01:07:20.130712 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.130684 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm"] Nov 01 01:07:20.167876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.167840 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8844a5a-492a-4534-a3af-07def8126146-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.167876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.167873 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8844a5a-492a-4534-a3af-07def8126146-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.167876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.167884 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.168116 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.167894 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsr9l\" (UniqueName: \"kubernetes.io/projected/d8844a5a-492a-4534-a3af-07def8126146-kube-api-access-nsr9l\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:20.177167 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.177137 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j"] Nov 01 01:07:20.354023 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.353934 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:20.867736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.867702 6036 patch_prober.go:29] interesting pod/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 01:07:20.868227 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.867756 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 01:07:20.893754 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.893717 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" Nov 01 01:07:20.893926 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.893780 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf" event={"ID":"d8844a5a-492a-4534-a3af-07def8126146","Type":"ContainerDied","Data":"f4795dec9ded84c72cfbab19e7dd9fd79ef3b1fc23d45bc49d24626e0cb7d016"} Nov 01 01:07:20.893926 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:20.893839 6036 scope.go:122] "RemoveContainer" containerID="56744e4edb65cc9e69c4feb326675cef32f08cc3d890faa600751d0b56023bc2" Nov 01 01:07:20.953000 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:20.952963 6036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8844a5a_492a_4534_a3af_07def8126146.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8844a5a_492a_4534_a3af_07def8126146.slice/crio-f4795dec9ded84c72cfbab19e7dd9fd79ef3b1fc23d45bc49d24626e0cb7d016\": RecentStats: unable to find data in memory cache]" Nov 01 01:07:21.063048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:21.063014 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf"] Nov 01 01:07:21.067607 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:21.067578 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-44c9bf-openshift-default-fcb47c9c9-wxdhf"] Nov 01 01:07:21.251523 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:21.251442 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8844a5a-492a-4534-a3af-07def8126146" path="/var/lib/kubelet/pods/d8844a5a-492a-4534-a3af-07def8126146/volumes" Nov 01 01:07:23.487972 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:23.487939 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:23.711720 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:23.711682 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:27.668382 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:27.668343 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:27.877930 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:27.877899 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:28.088409 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:28.088373 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:28.439279 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:28.439190 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:28.734518 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:28.734430 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:29.014771 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:29.014690 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:29.313747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:29.313712 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:30.336992 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:30.336957 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:30.701982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:30.701901 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:30.911783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:30.911736 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:31.317110 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:31.317066 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:33.626014 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:33.625971 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:33.860903 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:33.860858 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:34.095037 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:34.095003 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:35.148530 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.148492 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" containerID="cri-o://52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554" gracePeriod=45 Nov 01 01:07:35.152486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.152449 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" containerID="cri-o://3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded" gracePeriod=45 Nov 01 01:07:35.199543 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.199505 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" containerID="cri-o://42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea" gracePeriod=45 Nov 01 01:07:35.321081 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.321041 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:35.512476 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.512444 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 01:07:35.517432 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.517403 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 01:07:35.521280 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.521256 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 01:07:35.617026 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.616992 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-home\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-home\") pod \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " Nov 01 01:07:35.617211 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617088 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f488eb5a-65f0-4b05-8f36-a76f84a1937d-kube-api-access-r76g4\" (UniqueName: \"kubernetes.io/projected/f488eb5a-65f0-4b05-8f36-a76f84a1937d-kube-api-access-r76g4\") pod \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " Nov 01 01:07:35.617211 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617114 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-model-cache\") pod \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " Nov 01 01:07:35.617211 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617145 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tmp-dir\") pod \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " Nov 01 01:07:35.617411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617215 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-dshm\") pod \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " Nov 01 01:07:35.617411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617275 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-dshm\") pod \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " Nov 01 01:07:35.617411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617316 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-home" pod "6499de7b-330b-48bf-ae0a-86d6f1882c8c" (UID: "6499de7b-330b-48bf-ae0a-86d6f1882c8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.617411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617338 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-kube-api-access-6rrdv\" (UniqueName: \"kubernetes.io/projected/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-kube-api-access-6rrdv\") pod \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " Nov 01 01:07:35.617411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617370 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tls-certs\") pod \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " Nov 01 01:07:35.617411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617401 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tls-certs\") pod \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " Nov 01 01:07:35.617736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617495 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tmp-dir\") pod \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " Nov 01 01:07:35.617736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617491 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-model-cache" pod "f488eb5a-65f0-4b05-8f36-a76f84a1937d" (UID: "f488eb5a-65f0-4b05-8f36-a76f84a1937d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.617736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617539 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-dshm\") pod \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " Nov 01 01:07:35.617736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617574 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-home\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-home\") pod \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\" (UID: \"f488eb5a-65f0-4b05-8f36-a76f84a1937d\") " Nov 01 01:07:35.617736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617606 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tls-certs\") pod \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " Nov 01 01:07:35.617736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617661 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-model-cache\") pod \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " Nov 01 01:07:35.617736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617704 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tmp-dir\") pod \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " Nov 01 01:07:35.617736 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617738 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-model-cache\") pod \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " Nov 01 01:07:35.618166 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617767 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-home\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-home\") pod \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\" (UID: \"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9\") " Nov 01 01:07:35.618166 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617769 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tmp-dir" pod "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" (UID: "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.618166 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617798 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tmp-dir" pod "f488eb5a-65f0-4b05-8f36-a76f84a1937d" (UID: "f488eb5a-65f0-4b05-8f36-a76f84a1937d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.618166 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.617810 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6499de7b-330b-48bf-ae0a-86d6f1882c8c-kube-api-access-kvsd4\" (UniqueName: \"kubernetes.io/projected/6499de7b-330b-48bf-ae0a-86d6f1882c8c-kube-api-access-kvsd4\") pod \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\" (UID: \"6499de7b-330b-48bf-ae0a-86d6f1882c8c\") " Nov 01 01:07:35.618166 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618010 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-model-cache" pod "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" (UID: "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.618765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618740 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.619328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618769 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.619328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618786 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.619328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618800 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.619328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618813 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.619328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618766 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-model-cache" pod "6499de7b-330b-48bf-ae0a-86d6f1882c8c" (UID: "6499de7b-330b-48bf-ae0a-86d6f1882c8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.619328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618774 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-home" pod "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" (UID: "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.619328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618872 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-home" pod "f488eb5a-65f0-4b05-8f36-a76f84a1937d" (UID: "f488eb5a-65f0-4b05-8f36-a76f84a1937d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.619328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.618963 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tmp-dir" pod "6499de7b-330b-48bf-ae0a-86d6f1882c8c" (UID: "6499de7b-330b-48bf-ae0a-86d6f1882c8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.620768 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.620739 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f488eb5a-65f0-4b05-8f36-a76f84a1937d-kube-api-access-r76g4" pod "f488eb5a-65f0-4b05-8f36-a76f84a1937d" (UID: "f488eb5a-65f0-4b05-8f36-a76f84a1937d"). InnerVolumeSpecName "kube-api-access-r76g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:35.620915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.620885 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tls-certs" pod "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" (UID: "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:35.621001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.620916 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tls-certs" pod "f488eb5a-65f0-4b05-8f36-a76f84a1937d" (UID: "f488eb5a-65f0-4b05-8f36-a76f84a1937d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:35.621479 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.621435 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-kube-api-access-6rrdv" pod "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" (UID: "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9"). InnerVolumeSpecName "kube-api-access-6rrdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:35.621582 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.621552 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tls-certs" pod "6499de7b-330b-48bf-ae0a-86d6f1882c8c" (UID: "6499de7b-330b-48bf-ae0a-86d6f1882c8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:35.621758 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.621734 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-dshm" pod "6499de7b-330b-48bf-ae0a-86d6f1882c8c" (UID: "6499de7b-330b-48bf-ae0a-86d6f1882c8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.621935 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.621912 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-dshm" pod "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" (UID: "9e5fee26-de0f-43bb-b3b2-2a187e9b62a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.622191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.622175 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-dshm" pod "f488eb5a-65f0-4b05-8f36-a76f84a1937d" (UID: "f488eb5a-65f0-4b05-8f36-a76f84a1937d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.622956 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.622936 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6499de7b-330b-48bf-ae0a-86d6f1882c8c-kube-api-access-kvsd4" pod "6499de7b-330b-48bf-ae0a-86d6f1882c8c" (UID: "6499de7b-330b-48bf-ae0a-86d6f1882c8c"). InnerVolumeSpecName "kube-api-access-kvsd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:35.720877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720778 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.720877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720810 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.720877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720822 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.720877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720836 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.720877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720847 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499de7b-330b-48bf-ae0a-86d6f1882c8c-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.720877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720860 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.720877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720874 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvsd4\" (UniqueName: \"kubernetes.io/projected/6499de7b-330b-48bf-ae0a-86d6f1882c8c-kube-api-access-kvsd4\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.721231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720887 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r76g4\" (UniqueName: \"kubernetes.io/projected/f488eb5a-65f0-4b05-8f36-a76f84a1937d-kube-api-access-r76g4\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.721231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720900 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.721231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720911 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f488eb5a-65f0-4b05-8f36-a76f84a1937d-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.721231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720921 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rrdv\" (UniqueName: \"kubernetes.io/projected/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-kube-api-access-6rrdv\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.721231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720931 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.721231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.720942 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f488eb5a-65f0-4b05-8f36-a76f84a1937d-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.920344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.920250 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:35.994376 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.994250 6036 generic.go:356] "Generic (PLEG): container finished" podID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerID="42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea" exitCode=0 Nov 01 01:07:35.994376 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.994311 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" Nov 01 01:07:35.994376 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.994341 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" event={"ID":"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9","Type":"ContainerDied","Data":"42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea"} Nov 01 01:07:35.994667 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.994396 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j" event={"ID":"9e5fee26-de0f-43bb-b3b2-2a187e9b62a9","Type":"ContainerDied","Data":"fa9ae1c86afb525332ea9f7b2f2f2742a4048cd5551410aa80c081723a85e392"} Nov 01 01:07:35.994667 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.994421 6036 scope.go:122] "RemoveContainer" containerID="42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea" Nov 01 01:07:35.995957 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.995924 6036 generic.go:356] "Generic (PLEG): container finished" podID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerID="3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded" exitCode=0 Nov 01 01:07:35.996073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.995971 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" Nov 01 01:07:35.996073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.996012 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" event={"ID":"f488eb5a-65f0-4b05-8f36-a76f84a1937d","Type":"ContainerDied","Data":"3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded"} Nov 01 01:07:35.996073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.996052 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm" event={"ID":"f488eb5a-65f0-4b05-8f36-a76f84a1937d","Type":"ContainerDied","Data":"2b2382d0537feb851710d549368a6dc31dce2894547547af9db6ec09ab9d8760"} Nov 01 01:07:35.997674 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.997642 6036 generic.go:356] "Generic (PLEG): container finished" podID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerID="52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554" exitCode=0 Nov 01 01:07:35.997786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.997696 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" Nov 01 01:07:35.997786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.997709 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" event={"ID":"6499de7b-330b-48bf-ae0a-86d6f1882c8c","Type":"ContainerDied","Data":"52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554"} Nov 01 01:07:35.997786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:35.997745 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr" event={"ID":"6499de7b-330b-48bf-ae0a-86d6f1882c8c","Type":"ContainerDied","Data":"f6f199264cfaa2d4bd8f64acb587dd52154028c6d67f8c48bd233c047039c294"} Nov 01 01:07:36.005863 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.005846 6036 scope.go:122] "RemoveContainer" containerID="42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea" Nov 01 01:07:36.006263 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:36.006163 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea\": container with ID starting with 42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea not found: ID does not exist" containerID="42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea" Nov 01 01:07:36.006263 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.006192 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea"} err="failed to get container status \"42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea\": rpc error: code = NotFound desc = could not find container \"42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea\": container with ID starting with 42d9edbc42c5b71d71bca06ff4f90f1edc67b8687473c65fa074e0c419b654ea not found: ID does not exist" Nov 01 01:07:36.006263 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.006210 6036 scope.go:122] "RemoveContainer" containerID="3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded" Nov 01 01:07:36.017141 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.017116 6036 scope.go:122] "RemoveContainer" containerID="3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded" Nov 01 01:07:36.017507 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:36.017485 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded\": container with ID starting with 3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded not found: ID does not exist" containerID="3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded" Nov 01 01:07:36.017574 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.017513 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded"} err="failed to get container status \"3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded\": rpc error: code = NotFound desc = could not find container \"3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded\": container with ID starting with 3db7f2067e56d89a279d44418a67a58a83f299a88d2c7c63a2108d213415cded not found: ID does not exist" Nov 01 01:07:36.017574 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.017531 6036 scope.go:122] "RemoveContainer" containerID="52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554" Nov 01 01:07:36.029153 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.029129 6036 scope.go:122] "RemoveContainer" containerID="52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554" Nov 01 01:07:36.029544 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:36.029521 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554\": container with ID starting with 52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554 not found: ID does not exist" containerID="52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554" Nov 01 01:07:36.029637 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.029555 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554"} err="failed to get container status \"52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554\": rpc error: code = NotFound desc = could not find container \"52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554\": container with ID starting with 52e8269a6a096b46ccdd78da03d0d1e1fb53280048f81499c97841e102e90554 not found: ID does not exist" Nov 01 01:07:36.163945 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.163910 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j"] Nov 01 01:07:36.168406 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.168313 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/premium-w5-948e22-kserve-7d57d99467-n9l7j"] Nov 01 01:07:36.279544 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.279460 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm"] Nov 01 01:07:36.286061 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.286032 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-w5-948e22-kserve-d6fbc466b-vw7mm"] Nov 01 01:07:36.376196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.376162 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:36.400041 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.400004 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr"] Nov 01 01:07:36.406046 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.406018 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/distinct-2-w5-948e22-kserve-57bdd5ff7-7n7hr"] Nov 01 01:07:36.973712 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:36.973673 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:37.184556 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.184517 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:37.254596 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.254503 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" path="/var/lib/kubelet/pods/6499de7b-330b-48bf-ae0a-86d6f1882c8c/volumes" Nov 01 01:07:37.255125 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.255104 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" path="/var/lib/kubelet/pods/9e5fee26-de0f-43bb-b3b2-2a187e9b62a9/volumes" Nov 01 01:07:37.255638 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.255613 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" path="/var/lib/kubelet/pods/f488eb5a-65f0-4b05-8f36-a76f84a1937d/volumes" Nov 01 01:07:37.333724 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.333685 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv"] Nov 01 01:07:37.334659 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334626 6036 cpu_manager.go:414] "containerMap: removing container" podUID="050fffbe-b376-4cbc-ab7f-8b1c76349986" containerName="payload-pre-processing" Nov 01 01:07:37.334659 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334655 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050fffbe-b376-4cbc-ab7f-8b1c76349986" containerName="payload-pre-processing" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334684 6036 cpu_manager.go:414] "containerMap: removing container" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334693 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334713 6036 cpu_manager.go:414] "containerMap: removing container" podUID="f2fb595d-d9fe-40e8-9f09-8eea308bf3d7" containerName="payload-processing" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334721 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fb595d-d9fe-40e8-9f09-8eea308bf3d7" containerName="payload-processing" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334743 6036 cpu_manager.go:414] "containerMap: removing container" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334750 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334764 6036 cpu_manager.go:414] "containerMap: removing container" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334771 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334796 6036 cpu_manager.go:414] "containerMap: removing container" podUID="092dea27-ada1-4838-ab3b-a38b4dbc4d97" containerName="payload-pre-processing" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334804 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092dea27-ada1-4838-ab3b-a38b4dbc4d97" containerName="payload-pre-processing" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334817 6036 cpu_manager.go:414] "containerMap: removing container" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334824 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334838 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9c47c543-f8cd-4a7c-811f-2f884e97a10b" containerName="payload-processing" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334846 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c47c543-f8cd-4a7c-811f-2f884e97a10b" containerName="payload-processing" Nov 01 01:07:37.334853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334856 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.334864 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335005 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="050fffbe-b376-4cbc-ab7f-8b1c76349986" containerName="payload-pre-processing" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335020 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335030 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fb595d-d9fe-40e8-9f09-8eea308bf3d7" containerName="payload-processing" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335043 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f488eb5a-65f0-4b05-8f36-a76f84a1937d" containerName="main" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335053 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8844a5a-492a-4534-a3af-07def8126146" containerName="istio-proxy" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335075 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="092dea27-ada1-4838-ab3b-a38b4dbc4d97" containerName="payload-pre-processing" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335087 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6499de7b-330b-48bf-ae0a-86d6f1882c8c" containerName="main" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335098 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c47c543-f8cd-4a7c-811f-2f884e97a10b" containerName="payload-processing" Nov 01 01:07:37.335506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.335107 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5fee26-de0f-43bb-b3b2-2a187e9b62a9" containerName="main" Nov 01 01:07:37.362368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.357065 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:37.426358 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.426321 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv"] Nov 01 01:07:37.456646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.456610 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv\" (UID: \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:37.456889 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.456868 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrvp\" (UniqueName: \"kubernetes.io/projected/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-kube-api-access-mgrvp\") pod \"payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv\" (UID: \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:37.468015 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.467982 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cbea8a\"" Nov 01 01:07:37.468168 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.468076 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cbea8a-dockercfg-hj8bj\"" Nov 01 01:07:37.480077 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.480053 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm"] Nov 01 01:07:37.484172 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.484155 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:37.541534 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.541501 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm"] Nov 01 01:07:37.558246 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.558209 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnw2q\" (UniqueName: \"kubernetes.io/projected/ad50d0fc-648e-477a-9b20-1585dbce5592-kube-api-access-hnw2q\") pod \"payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm\" (UID: \"ad50d0fc-648e-477a-9b20-1585dbce5592\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:37.558437 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.558357 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrvp\" (UniqueName: \"kubernetes.io/projected/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-kube-api-access-mgrvp\") pod \"payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv\" (UID: \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:37.558560 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.558538 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv\" (UID: \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:37.558620 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.558608 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50d0fc-648e-477a-9b20-1585dbce5592-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm\" (UID: \"ad50d0fc-648e-477a-9b20-1585dbce5592\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:37.559265 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.559242 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv\" (UID: \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:37.620855 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.620822 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrvp\" (UniqueName: \"kubernetes.io/projected/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-kube-api-access-mgrvp\") pod \"payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv\" (UID: \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:37.659765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.659728 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50d0fc-648e-477a-9b20-1585dbce5592-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm\" (UID: \"ad50d0fc-648e-477a-9b20-1585dbce5592\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:37.659986 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.659844 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnw2q\" (UniqueName: \"kubernetes.io/projected/ad50d0fc-648e-477a-9b20-1585dbce5592-kube-api-access-hnw2q\") pod \"payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm\" (UID: \"ad50d0fc-648e-477a-9b20-1585dbce5592\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:37.660715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.660684 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50d0fc-648e-477a-9b20-1585dbce5592-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm\" (UID: \"ad50d0fc-648e-477a-9b20-1585dbce5592\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:37.722572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.722530 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnw2q\" (UniqueName: \"kubernetes.io/projected/ad50d0fc-648e-477a-9b20-1585dbce5592-kube-api-access-hnw2q\") pod \"payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm\" (UID: \"ad50d0fc-648e-477a-9b20-1585dbce5592\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:37.726272 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.726253 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:37.802140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:37.802059 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:38.107888 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:38.107863 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv"] Nov 01 01:07:38.108989 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:07:38.108966 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2997b6_e3bb_4a7a_81ce_dc62e780fc84.slice/crio-817151b5885907ff277ecae9617e3807893fc189077498a452f78f6ba1ed0a76 WatchSource:0}: Error finding container 817151b5885907ff277ecae9617e3807893fc189077498a452f78f6ba1ed0a76: Status 404 returned error can't find the container with id 817151b5885907ff277ecae9617e3807893fc189077498a452f78f6ba1ed0a76 Nov 01 01:07:38.208002 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:38.207973 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm"] Nov 01 01:07:38.208659 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:07:38.208637 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad50d0fc_648e_477a_9b20_1585dbce5592.slice/crio-cf2a077a07514dd9ec9dd4927b624a36b3fd3124e714023d0ce3c9c7c494b6c1 WatchSource:0}: Error finding container cf2a077a07514dd9ec9dd4927b624a36b3fd3124e714023d0ce3c9c7c494b6c1: Status 404 returned error can't find the container with id cf2a077a07514dd9ec9dd4927b624a36b3fd3124e714023d0ce3c9c7c494b6c1 Nov 01 01:07:39.023018 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:39.022977 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" event={"ID":"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84","Type":"ContainerStarted","Data":"d13f5d878836884bfa353f2ee31fbc9223d682776b83cd6afc6a7cb3be6be76d"} Nov 01 01:07:39.023237 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:39.023046 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" event={"ID":"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84","Type":"ContainerStarted","Data":"817151b5885907ff277ecae9617e3807893fc189077498a452f78f6ba1ed0a76"} Nov 01 01:07:39.023237 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:39.023082 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:39.024919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:39.024893 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" event={"ID":"ad50d0fc-648e-477a-9b20-1585dbce5592","Type":"ContainerStarted","Data":"7f77bce0d3ebc95667ced073d2040e7fae81cabc21eab7bd7a07607e44d95ac1"} Nov 01 01:07:39.025066 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:39.024936 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" event={"ID":"ad50d0fc-648e-477a-9b20-1585dbce5592","Type":"ContainerStarted","Data":"cf2a077a07514dd9ec9dd4927b624a36b3fd3124e714023d0ce3c9c7c494b6c1"} Nov 01 01:07:39.025596 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:39.025575 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:39.143760 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:39.143702 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" podStartSLOduration=2.143688186 podStartE2EDuration="2.143688186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:39.141916419 +0000 UTC m=+2604.017137435" watchObservedRunningTime="2026-11-01 01:07:39.143688186 +0000 UTC m=+2604.018909200" Nov 01 01:07:39.273946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:39.273806 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" podStartSLOduration=2.273723158 podStartE2EDuration="2.273723158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:39.26819732 +0000 UTC m=+2604.143418338" watchObservedRunningTime="2026-11-01 01:07:39.273723158 +0000 UTC m=+2604.148944191" Nov 01 01:07:41.162094 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:41.162052 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:45.508799 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.508758 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f"] Nov 01 01:07:45.512971 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.512955 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.523140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.523103 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:45.564418 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.564377 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.564674 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.564651 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.564823 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.564712 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.564823 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.564810 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.565100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.564991 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.565100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.565021 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.565271 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.565110 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hll\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-kube-api-access-r6hll\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.565271 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.565193 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6993831-97b9-40ab-aa44-964e2d96ad47-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.565271 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.565235 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.579177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.579134 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f"] Nov 01 01:07:45.666456 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666422 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.666456 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666461 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.666720 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666486 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.666720 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666531 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.666720 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666584 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.666720 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666705 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.666720 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666721 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.667017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666794 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hll\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-kube-api-access-r6hll\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.667017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666954 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.667017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.666980 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.667017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.667000 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6993831-97b9-40ab-aa44-964e2d96ad47-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.667211 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.667053 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.667556 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.667533 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.667847 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.667826 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6993831-97b9-40ab-aa44-964e2d96ad47-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.669005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.668982 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.669229 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.669208 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.729296 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.729245 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.729489 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.729468 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hll\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-kube-api-access-r6hll\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:45.833637 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:45.833598 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:46.298101 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:46.298063 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f"] Nov 01 01:07:46.334436 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:07:46.334406 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6993831_97b9_40ab_aa44_964e2d96ad47.slice/crio-645aacc619a6f3d89349c0f4bfdf21d4e95e920be18242dc5b6e1a4d10a0e308 WatchSource:0}: Error finding container 645aacc619a6f3d89349c0f4bfdf21d4e95e920be18242dc5b6e1a4d10a0e308: Status 404 returned error can't find the container with id 645aacc619a6f3d89349c0f4bfdf21d4e95e920be18242dc5b6e1a4d10a0e308 Nov 01 01:07:46.337106 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:46.337066 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:07:46.337218 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:46.337141 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:07:46.337218 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:46.337175 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:07:47.004309 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.003432 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg"] Nov 01 01:07:47.012160 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.010232 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.059326 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.059274 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg"] Nov 01 01:07:47.064918 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.064892 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-030da8-serving-cert\"" Nov 01 01:07:47.075968 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.075932 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:47.087910 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.087875 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" event={"ID":"d6993831-97b9-40ab-aa44-964e2d96ad47","Type":"ContainerStarted","Data":"b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756"} Nov 01 01:07:47.088094 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.087942 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" event={"ID":"d6993831-97b9-40ab-aa44-964e2d96ad47","Type":"ContainerStarted","Data":"645aacc619a6f3d89349c0f4bfdf21d4e95e920be18242dc5b6e1a4d10a0e308"} Nov 01 01:07:47.123614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.119359 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0685b340-bbbd-4d7b-874f-c3a1cacb0310-openshift-service-ca\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.123614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.119461 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqmh\" (UniqueName: \"kubernetes.io/projected/0685b340-bbbd-4d7b-874f-c3a1cacb0310-kube-api-access-bmqmh\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.123614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.119508 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-metrics-certs\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.123614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.119593 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.221501 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.221467 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0685b340-bbbd-4d7b-874f-c3a1cacb0310-openshift-service-ca\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.221703 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.221539 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqmh\" (UniqueName: \"kubernetes.io/projected/0685b340-bbbd-4d7b-874f-c3a1cacb0310-kube-api-access-bmqmh\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.221703 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.221582 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-metrics-certs\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.221703 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.221671 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.221878 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:47.221799 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-serving-cert: secret "maas-api-e2e-api-b-030da8-serving-cert" not found Nov 01 01:07:47.221878 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:47.221868 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls podName:0685b340-bbbd-4d7b-874f-c3a1cacb0310 nodeName:}" failed. No retries permitted until 2026-11-01 01:07:47.721847225 +0000 UTC m=+2612.597068222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls") pod "maas-api-e2e-api-b-030da8-5596d6d666-dckmg" (UID: "0685b340-bbbd-4d7b-874f-c3a1cacb0310") : secret "maas-api-e2e-api-b-030da8-serving-cert" not found Nov 01 01:07:47.222227 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.222205 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0685b340-bbbd-4d7b-874f-c3a1cacb0310-openshift-service-ca\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.224356 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.224328 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-metrics-certs\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.304202 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.304169 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqmh\" (UniqueName: \"kubernetes.io/projected/0685b340-bbbd-4d7b-874f-c3a1cacb0310-kube-api-access-bmqmh\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.304415 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.304247 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podStartSLOduration=2.303458134 podStartE2EDuration="2.303458134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:47.296241747 +0000 UTC m=+2612.171462777" watchObservedRunningTime="2026-11-01 01:07:47.303458134 +0000 UTC m=+2612.178679150" Nov 01 01:07:47.330766 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.330732 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:47.553329 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.551842 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:47.731502 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.731395 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.734427 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.734393 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls\") pod \"maas-api-e2e-api-b-030da8-5596d6d666-dckmg\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:47.785433 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.785399 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:47.835408 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.835364 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:47.837383 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.837353 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 01 01:07:47.837551 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.837405 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 01 01:07:47.943576 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:47.943542 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:48.023250 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:48.022908 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:48.373324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:48.373258 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg"] Nov 01 01:07:48.373765 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:07:48.373735 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0685b340_bbbd_4d7b_874f_c3a1cacb0310.slice/crio-d54e1f98681d9ce3f4db7ac350d03514775dec5d2a8877684024756bfee1737e WatchSource:0}: Error finding container d54e1f98681d9ce3f4db7ac350d03514775dec5d2a8877684024756bfee1737e: Status 404 returned error can't find the container with id d54e1f98681d9ce3f4db7ac350d03514775dec5d2a8877684024756bfee1737e Nov 01 01:07:48.376528 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:48.376508 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 01:07:48.835740 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:48.835706 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 01 01:07:48.835934 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:48.835772 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 01 01:07:49.103009 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:49.102927 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" event={"ID":"0685b340-bbbd-4d7b-874f-c3a1cacb0310","Type":"ContainerStarted","Data":"d54e1f98681d9ce3f4db7ac350d03514775dec5d2a8877684024756bfee1737e"} Nov 01 01:07:49.515207 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:07:49.515121 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:07:49.835557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:49.835522 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 01 01:07:49.835761 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:49.835578 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 01 01:07:50.034023 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:50.033991 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:07:50.112425 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:50.112326 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" event={"ID":"0685b340-bbbd-4d7b-874f-c3a1cacb0310","Type":"ContainerStarted","Data":"2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc"} Nov 01 01:07:50.112881 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:50.112474 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:50.290482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:50.290433 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" podStartSLOduration=3.454968796 podStartE2EDuration="4.290418913s" totalImagesPullingTime="835.450117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:50.288348434 +0000 UTC m=+2615.163569450" watchObservedRunningTime="2026-11-01 01:07:50.290418913 +0000 UTC m=+2615.165639927" Nov 01 01:07:50.835519 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:50.835480 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 01 01:07:50.835714 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:50.835542 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 01 01:07:51.041193 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:51.041157 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:07:51.835619 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:51.835579 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 01 01:07:51.836037 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:51.835643 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 01 01:07:52.793672 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:52.793638 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:52.834945 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:52.834911 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 01 01:07:52.835131 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:52.834969 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 01 01:07:54.181173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:54.181081 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:54.181843 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:54.181812 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:54.183024 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:54.183000 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:07:54.490352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:54.489262 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:54.710201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:54.710159 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:54.931367 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:54.929967 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:55.423008 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:55.422968 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:56.125026 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:56.124996 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:07:57.481807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:57.481768 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:58.214979 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:58.214942 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:58.428439 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:58.428406 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:58.681856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:58.681821 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:59.305928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:59.305885 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:59.640114 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:59.640021 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:07:59.856471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:07:59.856432 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:00.333975 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:00.333939 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns"] Nov 01 01:08:00.385147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:00.385114 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8"] Nov 01 01:08:00.414003 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:00.413966 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:00.622017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:00.621924 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:04.635738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:04.635701 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:04.953993 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:04.953898 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:06.593137 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:06.593099 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:06.814622 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:06.814581 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:06.883080 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:06.882997 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q"] Nov 01 01:08:06.883242 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:06.883196 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" containerID="cri-o://8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d" gracePeriod=30 Nov 01 01:08:06.898725 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:06.898694 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f"] Nov 01 01:08:06.898924 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:06.898902 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" containerID="cri-o://b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756" gracePeriod=30 Nov 01 01:08:07.040140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:07.040100 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:07.935741 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:07.935701 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:08.154636 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:08.154600 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:09.190017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:09.189967 6036 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:08:09.190888 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:09.190844 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:08:10.498797 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:10.498762 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:10.714118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:10.714082 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:12.191504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.191472 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 01:08:12.225537 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.225510 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:08:12.264409 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.264373 6036 generic.go:356] "Generic (PLEG): container finished" podID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerID="b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756" exitCode=0 Nov 01 01:08:12.264609 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.264414 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" event={"ID":"d6993831-97b9-40ab-aa44-964e2d96ad47","Type":"ContainerDied","Data":"b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756"} Nov 01 01:08:12.264609 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.264489 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" event={"ID":"d6993831-97b9-40ab-aa44-964e2d96ad47","Type":"ContainerDied","Data":"645aacc619a6f3d89349c0f4bfdf21d4e95e920be18242dc5b6e1a4d10a0e308"} Nov 01 01:08:12.264609 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.264425 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f" Nov 01 01:08:12.264609 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.264524 6036 scope.go:122] "RemoveContainer" containerID="b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756" Nov 01 01:08:12.266075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.266051 6036 generic.go:356] "Generic (PLEG): container finished" podID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerID="8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d" exitCode=0 Nov 01 01:08:12.266190 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.266119 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" event={"ID":"0a2d1e3f-03be-449c-be85-7b2f090b98f4","Type":"ContainerDied","Data":"8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d"} Nov 01 01:08:12.266190 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.266158 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" event={"ID":"0a2d1e3f-03be-449c-be85-7b2f090b98f4","Type":"ContainerDied","Data":"9207065b386706f812f63a15114f733d1262d150ec145b9fd6edd38dffa0823c"} Nov 01 01:08:12.266341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.266218 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q" Nov 01 01:08:12.278090 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.278064 6036 scope.go:122] "RemoveContainer" containerID="b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756" Nov 01 01:08:12.278469 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:12.278447 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756\": container with ID starting with b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756 not found: ID does not exist" containerID="b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756" Nov 01 01:08:12.278577 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.278479 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756"} err="failed to get container status \"b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756\": rpc error: code = NotFound desc = could not find container \"b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756\": container with ID starting with b8c67ac731504946ef6c691945fc925de8199fffbb3d0eaffa7e2de768a68756 not found: ID does not exist" Nov 01 01:08:12.278577 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.278498 6036 scope.go:122] "RemoveContainer" containerID="8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d" Nov 01 01:08:12.288178 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.288160 6036 scope.go:122] "RemoveContainer" containerID="8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d" Nov 01 01:08:12.288493 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:12.288470 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d\": container with ID starting with 8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d not found: ID does not exist" containerID="8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d" Nov 01 01:08:12.288589 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.288504 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d"} err="failed to get container status \"8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d\": rpc error: code = NotFound desc = could not find container \"8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d\": container with ID starting with 8aa9fbb428866a44fe64c3c0e3b8d51e9cff305d4e36f34da2540092fe95717d not found: ID does not exist" Nov 01 01:08:12.290972 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.290953 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-data\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.290998 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-certs\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291059 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-envoy\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.291189 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291102 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-data\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.291189 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291143 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-certs\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.291317 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291184 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-podinfo\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291317 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291236 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-socket\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291426 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291326 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-credential-socket\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291426 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291363 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-credential-socket\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.291426 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291370 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-certs" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.291597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291427 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-kube-api-access-r6hll\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-kube-api-access-r6hll\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.291597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291454 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-certs" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.291597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291467 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-token\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-token\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.291597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291484 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-data" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.291597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291499 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6993831-97b9-40ab-aa44-964e2d96ad47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6993831-97b9-40ab-aa44-964e2d96ad47-istiod-ca-cert\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.291597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291550 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istiod-ca-cert\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291576 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-data" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.291973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291601 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-credential-socket" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.291973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291647 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-token\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-token\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291706 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-socket\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.291973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291763 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-kube-api-access-2txsf\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-kube-api-access-2txsf\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291795 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-socket" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.291973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291796 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-envoy\") pod \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\" (UID: \"0a2d1e3f-03be-449c-be85-7b2f090b98f4\") " Nov 01 01:08:12.291973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291856 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-credential-socket" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.291973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.291899 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-podinfo\") pod \"d6993831-97b9-40ab-aa44-964e2d96ad47\" (UID: \"d6993831-97b9-40ab-aa44-964e2d96ad47\") " Nov 01 01:08:12.292388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292371 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-socket" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.292919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292781 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.292919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292807 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.292919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292822 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.292919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292839 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.292919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292853 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.292919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292867 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.292919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292880 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.292919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.292893 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.293228 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.293032 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6993831-97b9-40ab-aa44-964e2d96ad47-istiod-ca-cert" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:12.293493 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.293452 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istiod-ca-cert" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:12.294640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.294609 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-envoy" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.294891 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.294872 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-podinfo" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:08:12.295084 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.295046 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-podinfo" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:08:12.295816 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.295788 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-kube-api-access-2txsf" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "kube-api-access-2txsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:12.296140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.296117 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-token" pod "0a2d1e3f-03be-449c-be85-7b2f090b98f4" (UID: "0a2d1e3f-03be-449c-be85-7b2f090b98f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:12.296256 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.296239 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-kube-api-access-r6hll" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "kube-api-access-r6hll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:12.296445 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.296431 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-envoy" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.296645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.296632 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-token" pod "d6993831-97b9-40ab-aa44-964e2d96ad47" (UID: "d6993831-97b9-40ab-aa44-964e2d96ad47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:12.395145 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395102 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395145 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395144 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395160 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6hll\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-kube-api-access-r6hll\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395177 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395192 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6993831-97b9-40ab-aa44-964e2d96ad47-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395205 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395218 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395232 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txsf\" (UniqueName: \"kubernetes.io/projected/0a2d1e3f-03be-449c-be85-7b2f090b98f4-kube-api-access-2txsf\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395246 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2d1e3f-03be-449c-be85-7b2f090b98f4-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.395471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.395258 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6993831-97b9-40ab-aa44-964e2d96ad47-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.692000 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.691774 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f"] Nov 01 01:08:12.696510 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.696480 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-m2q5f"] Nov 01 01:08:12.808654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.808620 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q"] Nov 01 01:08:12.817675 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.817642 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-snr7q"] Nov 01 01:08:12.950357 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:12.950324 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:13.256442 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:13.256319 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" path="/var/lib/kubelet/pods/0a2d1e3f-03be-449c-be85-7b2f090b98f4/volumes" Nov 01 01:08:13.257080 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:13.257056 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" path="/var/lib/kubelet/pods/d6993831-97b9-40ab-aa44-964e2d96ad47/volumes" Nov 01 01:08:13.868595 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:13.868545 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:15.356940 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.356877 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" containerID="cri-o://6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec" gracePeriod=45 Nov 01 01:08:15.383339 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.383273 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:15.411053 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.411013 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" containerID="cri-o://4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71" gracePeriod=45 Nov 01 01:08:15.485431 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.485393 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": dial tcp 10.134.0.66:8000: connect: connection refused" Nov 01 01:08:15.698734 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.698705 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 01:08:15.701715 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.701689 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 01:08:15.853364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853328 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-model-cache\") pod \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " Nov 01 01:08:15.853533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853393 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tmp-dir\") pod \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " Nov 01 01:08:15.853533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853434 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-model-cache\") pod \"477b9668-6573-401c-b659-5feeec968d5a\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " Nov 01 01:08:15.853533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853470 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-dshm\") pod \"477b9668-6573-401c-b659-5feeec968d5a\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " Nov 01 01:08:15.853718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853545 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/477b9668-6573-401c-b659-5feeec968d5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/477b9668-6573-401c-b659-5feeec968d5a-tls-certs\") pod \"477b9668-6573-401c-b659-5feeec968d5a\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " Nov 01 01:08:15.853718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853607 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-home\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-home\") pod \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " Nov 01 01:08:15.853718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853653 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-dshm\") pod \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " Nov 01 01:08:15.853877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853760 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-home\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-home\") pod \"477b9668-6573-401c-b659-5feeec968d5a\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " Nov 01 01:08:15.853877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853806 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-tmp-dir\") pod \"477b9668-6573-401c-b659-5feeec968d5a\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " Nov 01 01:08:15.853999 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853886 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-kube-api-access-6skcr\" (UniqueName: \"kubernetes.io/projected/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-kube-api-access-6skcr\") pod \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " Nov 01 01:08:15.853999 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853920 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477b9668-6573-401c-b659-5feeec968d5a-kube-api-access-7bh9b\" (UniqueName: \"kubernetes.io/projected/477b9668-6573-401c-b659-5feeec968d5a-kube-api-access-7bh9b\") pod \"477b9668-6573-401c-b659-5feeec968d5a\" (UID: \"477b9668-6573-401c-b659-5feeec968d5a\") " Nov 01 01:08:15.853999 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853953 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tls-certs\") pod \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\" (UID: \"6d5018c8-eafb-4c4f-87a5-9035c638eaf1\") " Nov 01 01:08:15.853999 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.853978 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-home" pod "477b9668-6573-401c-b659-5feeec968d5a" (UID: "477b9668-6573-401c-b659-5feeec968d5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.854204 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854077 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tmp-dir" pod "6d5018c8-eafb-4c4f-87a5-9035c638eaf1" (UID: "6d5018c8-eafb-4c4f-87a5-9035c638eaf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.854204 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854084 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-model-cache" pod "477b9668-6573-401c-b659-5feeec968d5a" (UID: "477b9668-6573-401c-b659-5feeec968d5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.854583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854554 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-home" pod "6d5018c8-eafb-4c4f-87a5-9035c638eaf1" (UID: "6d5018c8-eafb-4c4f-87a5-9035c638eaf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.855019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854752 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-tmp-dir" pod "477b9668-6573-401c-b659-5feeec968d5a" (UID: "477b9668-6573-401c-b659-5feeec968d5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.855019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854924 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.855019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854942 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.855019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854957 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.855019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854970 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.855019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.854983 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.855548 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.855528 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-model-cache" pod "6d5018c8-eafb-4c4f-87a5-9035c638eaf1" (UID: "6d5018c8-eafb-4c4f-87a5-9035c638eaf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.858068 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.858006 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-kube-api-access-6skcr" pod "6d5018c8-eafb-4c4f-87a5-9035c638eaf1" (UID: "6d5018c8-eafb-4c4f-87a5-9035c638eaf1"). InnerVolumeSpecName "kube-api-access-6skcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:15.858377 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.858094 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-dshm" pod "6d5018c8-eafb-4c4f-87a5-9035c638eaf1" (UID: "6d5018c8-eafb-4c4f-87a5-9035c638eaf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.858377 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.858151 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477b9668-6573-401c-b659-5feeec968d5a-tls-certs" pod "477b9668-6573-401c-b659-5feeec968d5a" (UID: "477b9668-6573-401c-b659-5feeec968d5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:08:15.858907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.858863 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-dshm" pod "477b9668-6573-401c-b659-5feeec968d5a" (UID: "477b9668-6573-401c-b659-5feeec968d5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.859076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.858959 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tls-certs" pod "6d5018c8-eafb-4c4f-87a5-9035c638eaf1" (UID: "6d5018c8-eafb-4c4f-87a5-9035c638eaf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:08:15.859815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.859783 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477b9668-6573-401c-b659-5feeec968d5a-kube-api-access-7bh9b" pod "477b9668-6573-401c-b659-5feeec968d5a" (UID: "477b9668-6573-401c-b659-5feeec968d5a"). InnerVolumeSpecName "kube-api-access-7bh9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:15.956698 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.956664 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477b9668-6573-401c-b659-5feeec968d5a-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.956698 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.956701 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.956901 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.956717 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6skcr\" (UniqueName: \"kubernetes.io/projected/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-kube-api-access-6skcr\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.956901 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.956731 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bh9b\" (UniqueName: \"kubernetes.io/projected/477b9668-6573-401c-b659-5feeec968d5a-kube-api-access-7bh9b\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.956901 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.956745 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.956901 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.956759 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5018c8-eafb-4c4f-87a5-9035c638eaf1-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.956901 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:15.956771 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477b9668-6573-401c-b659-5feeec968d5a-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:16.295664 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.295623 6036 generic.go:356] "Generic (PLEG): container finished" podID="477b9668-6573-401c-b659-5feeec968d5a" containerID="4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71" exitCode=0 Nov 01 01:08:16.295827 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.295682 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" Nov 01 01:08:16.295827 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.295715 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" event={"ID":"477b9668-6573-401c-b659-5feeec968d5a","Type":"ContainerDied","Data":"4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71"} Nov 01 01:08:16.295827 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.295765 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8" event={"ID":"477b9668-6573-401c-b659-5feeec968d5a","Type":"ContainerDied","Data":"f25aea948e62615cd42f27c57ae0509b8013b429b9f2c2d65adffd1a867bfdf6"} Nov 01 01:08:16.295827 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.295788 6036 scope.go:122] "RemoveContainer" containerID="4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71" Nov 01 01:08:16.297548 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.297518 6036 generic.go:356] "Generic (PLEG): container finished" podID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerID="6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec" exitCode=0 Nov 01 01:08:16.297654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.297587 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" event={"ID":"6d5018c8-eafb-4c4f-87a5-9035c638eaf1","Type":"ContainerDied","Data":"6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec"} Nov 01 01:08:16.297654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.297621 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" event={"ID":"6d5018c8-eafb-4c4f-87a5-9035c638eaf1","Type":"ContainerDied","Data":"5bc6a03b29907a695800556c75d03b798da1006fcfb2632a8034e9e4603a3a89"} Nov 01 01:08:16.297787 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.297694 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns" Nov 01 01:08:16.308004 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.307981 6036 scope.go:122] "RemoveContainer" containerID="4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71" Nov 01 01:08:16.308410 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:16.308384 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71\": container with ID starting with 4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71 not found: ID does not exist" containerID="4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71" Nov 01 01:08:16.308470 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.308422 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71"} err="failed to get container status \"4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71\": rpc error: code = NotFound desc = could not find container \"4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71\": container with ID starting with 4c7379656516aacb329db9084847a85ec5aed9a323f40917655db3faa612fb71 not found: ID does not exist" Nov 01 01:08:16.308470 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.308441 6036 scope.go:122] "RemoveContainer" containerID="6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec" Nov 01 01:08:16.317485 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.317462 6036 scope.go:122] "RemoveContainer" containerID="6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec" Nov 01 01:08:16.317766 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:16.317745 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec\": container with ID starting with 6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec not found: ID does not exist" containerID="6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec" Nov 01 01:08:16.317847 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.317779 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec"} err="failed to get container status \"6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec\": rpc error: code = NotFound desc = could not find container \"6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec\": container with ID starting with 6313ed2bd38f0e65a0010b2fa49c39708e7aab4c5a1b9bf45864d1da33eba8ec not found: ID does not exist" Nov 01 01:08:16.465257 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.465218 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8"] Nov 01 01:08:16.516087 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.516054 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/simulator-w0-32439f-kserve-77dfd67cdb-h44p8"] Nov 01 01:08:16.628355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.628261 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns"] Nov 01 01:08:16.642313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:16.642247 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-2-w0-32439f-kserve-79f769c4d5-zzgns"] Nov 01 01:08:17.251312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:17.251255 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477b9668-6573-401c-b659-5feeec968d5a" path="/var/lib/kubelet/pods/477b9668-6573-401c-b659-5feeec968d5a/volumes" Nov 01 01:08:17.251808 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:17.251794 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" path="/var/lib/kubelet/pods/6d5018c8-eafb-4c4f-87a5-9035c638eaf1/volumes" Nov 01 01:08:19.571248 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:19.571201 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:08:19.607583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:19.607557 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:19.820354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:19.820315 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:21.214450 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:21.214405 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:21.438156 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:21.438107 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:21.653129 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:21.653095 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:22.194482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:22.194434 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:22.511055 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:22.510965 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:23.223140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:23.223101 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:23.855796 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:23.855753 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:24.191518 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:24.191440 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:26.340198 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340162 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2"] Nov 01 01:08:26.340788 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340774 6036 cpu_manager.go:414] "containerMap: removing container" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" Nov 01 01:08:26.340788 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340789 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" Nov 01 01:08:26.340873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340820 6036 cpu_manager.go:414] "containerMap: removing container" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" Nov 01 01:08:26.340873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340825 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" Nov 01 01:08:26.340873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340838 6036 cpu_manager.go:414] "containerMap: removing container" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" Nov 01 01:08:26.340873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340843 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" Nov 01 01:08:26.340873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340853 6036 cpu_manager.go:414] "containerMap: removing container" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" Nov 01 01:08:26.340873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340859 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" Nov 01 01:08:26.341093 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340945 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5018c8-eafb-4c4f-87a5-9035c638eaf1" containerName="main" Nov 01 01:08:26.341093 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340958 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="477b9668-6573-401c-b659-5feeec968d5a" containerName="main" Nov 01 01:08:26.341093 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340965 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6993831-97b9-40ab-aa44-964e2d96ad47" containerName="istio-proxy" Nov 01 01:08:26.341093 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.340972 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2d1e3f-03be-449c-be85-7b2f090b98f4" containerName="istio-proxy" Nov 01 01:08:26.346169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.346150 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.376105 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.376065 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:26.450168 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.450120 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-metrics-certs\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.450356 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.450298 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d1a84-c47b-4c0e-b994-e26325e429cd-openshift-service-ca\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.450521 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.450506 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.450602 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.450581 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6zn6\" (UniqueName: \"kubernetes.io/projected/fd1d1a84-c47b-4c0e-b994-e26325e429cd-kube-api-access-t6zn6\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.452336 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.452313 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cbea8a-serving-cert\"" Nov 01 01:08:26.455355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.455332 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2"] Nov 01 01:08:26.551889 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.551858 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d1a84-c47b-4c0e-b994-e26325e429cd-openshift-service-ca\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.552071 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.551996 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.552071 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.552025 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6zn6\" (UniqueName: \"kubernetes.io/projected/fd1d1a84-c47b-4c0e-b994-e26325e429cd-kube-api-access-t6zn6\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.552178 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.552076 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-metrics-certs\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.552178 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:26.552129 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-serving-cert: secret "maas-api-e2e-api-b-cbea8a-serving-cert" not found Nov 01 01:08:26.552257 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:26.552199 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls podName:fd1d1a84-c47b-4c0e-b994-e26325e429cd nodeName:}" failed. No retries permitted until 2026-11-01 01:08:27.052178542 +0000 UTC m=+2651.927399536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls") pod "maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" (UID: "fd1d1a84-c47b-4c0e-b994-e26325e429cd") : secret "maas-api-e2e-api-b-cbea8a-serving-cert" not found Nov 01 01:08:26.552678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.552651 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d1a84-c47b-4c0e-b994-e26325e429cd-openshift-service-ca\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.555159 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.555137 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-metrics-certs\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.584966 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.584936 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:26.616025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.615941 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6zn6\" (UniqueName: \"kubernetes.io/projected/fd1d1a84-c47b-4c0e-b994-e26325e429cd-kube-api-access-t6zn6\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:26.813445 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:26.807427 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:27.015837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:27.015741 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:27.059671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:27.059633 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:27.062380 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:27.062353 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls\") pod \"maas-api-e2e-api-b-cbea8a-784947f494-hnxh2\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:27.260562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:27.260532 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:27.640735 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:27.640699 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2"] Nov 01 01:08:27.641814 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:08:27.641772 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1d1a84_c47b_4c0e_b994_e26325e429cd.slice/crio-03ca1e06d048b60f86b90c6708749fa73f328665f61f302f88f440aa5e205750 WatchSource:0}: Error finding container 03ca1e06d048b60f86b90c6708749fa73f328665f61f302f88f440aa5e205750: Status 404 returned error can't find the container with id 03ca1e06d048b60f86b90c6708749fa73f328665f61f302f88f440aa5e205750 Nov 01 01:08:28.294214 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:28.294177 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:28.373971 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:28.373930 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" event={"ID":"fd1d1a84-c47b-4c0e-b994-e26325e429cd","Type":"ContainerStarted","Data":"03ca1e06d048b60f86b90c6708749fa73f328665f61f302f88f440aa5e205750"} Nov 01 01:08:28.770912 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:28.770874 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:28.856092 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:28.856008 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq"] Nov 01 01:08:28.856247 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:28.856223 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" containerID="cri-o://a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875" gracePeriod=30 Nov 01 01:08:29.000780 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:29.000741 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:29.381402 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:29.381366 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" event={"ID":"fd1d1a84-c47b-4c0e-b994-e26325e429cd","Type":"ContainerStarted","Data":"7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f"} Nov 01 01:08:29.381601 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:29.381583 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:29.502658 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:29.502604 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" podStartSLOduration=2.643280973 podStartE2EDuration="3.502588892s" totalImagesPullingTime="859.307919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:08:29.49984633 +0000 UTC m=+2654.375067391" watchObservedRunningTime="2026-11-01 01:08:29.502588892 +0000 UTC m=+2654.377809907" Nov 01 01:08:31.508882 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:31.508847 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:31.605964 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:31.605910 6036 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:08:31.606505 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:31.606267 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:08:31.760422 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:31.752821 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:32.321011 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:32.320962 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:34.143760 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.143733 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 01:08:34.220393 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220351 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istiod-ca-cert\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.220573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220474 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-envoy\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.220573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220503 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-data\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.220573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220534 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-token\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-token\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.220765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220579 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-socket\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.220765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220667 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-podinfo\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.220765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220693 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-certs\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.220921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220811 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-kube-api-access-g2css\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-kube-api-access-g2css\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.220921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220808 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-data" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.220921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220835 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istiod-ca-cert" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:34.220921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.220855 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-credential-socket\") pod \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\" (UID: \"5e4a2c30-4b8f-4d6e-90f8-8569f072d908\") " Nov 01 01:08:34.221129 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.221029 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-socket" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.221190 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.221142 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-certs" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.221335 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.221315 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-credential-socket" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.221634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.221615 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.221719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.221638 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.221719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.221652 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.221719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.221665 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.221719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.221677 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.222948 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.222922 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-token" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:34.223099 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.223078 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-kube-api-access-g2css" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "kube-api-access-g2css". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:34.223555 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.223526 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-podinfo" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:08:34.223555 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.223542 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-envoy" pod "5e4a2c30-4b8f-4d6e-90f8-8569f072d908" (UID: "5e4a2c30-4b8f-4d6e-90f8-8569f072d908"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.325514 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.325477 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.325803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.325771 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.325997 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.325982 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.326090 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.326078 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2css\" (UniqueName: \"kubernetes.io/projected/5e4a2c30-4b8f-4d6e-90f8-8569f072d908-kube-api-access-g2css\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.419991 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.419895 6036 generic.go:356] "Generic (PLEG): container finished" podID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerID="a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875" exitCode=0 Nov 01 01:08:34.419991 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.419959 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" Nov 01 01:08:34.420231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.419984 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" event={"ID":"5e4a2c30-4b8f-4d6e-90f8-8569f072d908","Type":"ContainerDied","Data":"a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875"} Nov 01 01:08:34.420231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.420076 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq" event={"ID":"5e4a2c30-4b8f-4d6e-90f8-8569f072d908","Type":"ContainerDied","Data":"932c8a101da903f09af7032e351cba23926de2298a05191b59072a3372193c7c"} Nov 01 01:08:34.420231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.420107 6036 scope.go:122] "RemoveContainer" containerID="a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875" Nov 01 01:08:34.434970 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.434945 6036 scope.go:122] "RemoveContainer" containerID="73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091" Nov 01 01:08:34.444528 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.444509 6036 scope.go:122] "RemoveContainer" containerID="a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875" Nov 01 01:08:34.444809 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:34.444791 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875\": container with ID starting with a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875 not found: ID does not exist" containerID="a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875" Nov 01 01:08:34.444863 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.444819 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875"} err="failed to get container status \"a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875\": rpc error: code = NotFound desc = could not find container \"a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875\": container with ID starting with a2e6052568d8c359dfd5752ffeee1aa06ff271cfef4e5a06b59588fb707c4875 not found: ID does not exist" Nov 01 01:08:34.444863 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.444836 6036 scope.go:122] "RemoveContainer" containerID="73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091" Nov 01 01:08:34.445097 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:34.445077 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091\": container with ID starting with 73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091 not found: ID does not exist" containerID="73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091" Nov 01 01:08:34.445142 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.445106 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091"} err="failed to get container status \"73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091\": rpc error: code = NotFound desc = could not find container \"73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091\": container with ID starting with 73687399caac26d8f9942af56f479f838a1a00b4e85007113f56159eef504091 not found: ID does not exist" Nov 01 01:08:34.587738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.587675 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq"] Nov 01 01:08:34.591582 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:34.591555 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-bqxwq"] Nov 01 01:08:35.252171 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:35.252136 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" path="/var/lib/kubelet/pods/5e4a2c30-4b8f-4d6e-90f8-8569f072d908/volumes" Nov 01 01:08:35.393092 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:35.393064 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:08:35.673902 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:35.673869 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:37.590274 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:37.590240 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:38.003989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:38.003903 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:38.333712 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:38.333650 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:39.104774 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:39.104740 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:42.620067 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:42.620029 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:45.538246 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:45.538208 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:47.542344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:47.542307 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:48.403565 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:48.403524 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:49.505950 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:08:49.505913 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:08:52.821958 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:52.821920 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:08:58.090353 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:08:58.090254 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:00.055981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.055936 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:00.265062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.265031 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:00.481645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.481560 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2"] Nov 01 01:09:00.482224 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.482208 6036 cpu_manager.go:414] "containerMap: removing container" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" Nov 01 01:09:00.482323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.482226 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" Nov 01 01:09:00.482435 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.482419 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" Nov 01 01:09:00.486311 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.486269 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.579475 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.579442 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2"] Nov 01 01:09:00.609886 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.609849 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.609886 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.609897 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crj9b\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-kube-api-access-crj9b\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.610141 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.609935 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.610141 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.610070 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb59fb86-48cd-448e-9d57-66f80ee699c6-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.610253 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.610201 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.610403 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.610379 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.610474 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.610432 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.610474 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.610468 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.610572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.610543 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.711566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.711523 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.711566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.711572 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.711803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.711594 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.711803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.711648 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.711803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.711694 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.711972 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.711910 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crj9b\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-kube-api-access-crj9b\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.712136 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.712086 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.712136 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.712119 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.712328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.712252 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.712395 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.712368 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb59fb86-48cd-448e-9d57-66f80ee699c6-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.712500 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.712478 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.712729 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.712554 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.712935 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.712893 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.713026 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.712973 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb59fb86-48cd-448e-9d57-66f80ee699c6-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.714393 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.714372 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.714699 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.714676 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.773937 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.773865 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.773937 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.773929 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crj9b\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-kube-api-access-crj9b\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:00.804570 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:00.804529 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:01.115477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.115441 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r"] Nov 01 01:09:01.228413 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.228379 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b"] Nov 01 01:09:01.240883 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:09:01.240855 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb59fb86_48cd_448e_9d57_66f80ee699c6.slice/crio-bd07df52d56645bc6a34989e176b3be533f65874fbf37e17fbd2467f552fc289 WatchSource:0}: Error finding container bd07df52d56645bc6a34989e176b3be533f65874fbf37e17fbd2467f552fc289: Status 404 returned error can't find the container with id bd07df52d56645bc6a34989e176b3be533f65874fbf37e17fbd2467f552fc289 Nov 01 01:09:01.242470 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.242445 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2"] Nov 01 01:09:01.243782 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.243747 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:01.243874 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.243814 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:01.243874 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.243843 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:01.575354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.575253 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:01.580322 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.580275 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd"] Nov 01 01:09:01.580762 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.580732 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" containerID="cri-o://1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c" gracePeriod=30 Nov 01 01:09:01.584583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.584548 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4"] Nov 01 01:09:01.584881 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.584834 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" containerID="cri-o://b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af" gracePeriod=30 Nov 01 01:09:01.589146 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.589113 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g"] Nov 01 01:09:01.589384 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.589359 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" containerID="cri-o://a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93" gracePeriod=30 Nov 01 01:09:01.610578 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.610539 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" event={"ID":"cb59fb86-48cd-448e-9d57-66f80ee699c6","Type":"ContainerStarted","Data":"22159a08fa8e038360b70f720ee88459c65596a294cee53f156cc1e39bbe59ce"} Nov 01 01:09:01.610752 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.610607 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" event={"ID":"cb59fb86-48cd-448e-9d57-66f80ee699c6","Type":"ContainerStarted","Data":"bd07df52d56645bc6a34989e176b3be533f65874fbf37e17fbd2467f552fc289"} Nov 01 01:09:01.792993 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.792963 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:01.804928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.804898 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:01.973231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.973102 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" podStartSLOduration=1.97308171 podStartE2EDuration="1.97308171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:09:01.971819933 +0000 UTC m=+2686.847040948" watchObservedRunningTime="2026-11-01 01:09:01.97308171 +0000 UTC m=+2686.848302727" Nov 01 01:09:01.974831 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.974799 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:09:01.974994 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:01.974864 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:09:02.031375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:02.031330 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:09:02.031542 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:02.031395 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:09:02.171588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:02.171536 6036 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:09:02.172092 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:02.171598 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:09:02.488878 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:02.488833 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:02.732738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:02.732698 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:02.807379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:02.807344 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 01:09:02.807583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:02.807400 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 01:09:03.695263 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:03.695217 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:03.805803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:03.805773 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 01:09:03.805983 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:03.805826 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 01:09:04.805538 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:04.805499 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 01:09:04.805969 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:04.805553 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 01:09:05.805021 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:05.804974 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 01:09:05.805234 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:05.805030 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 01:09:06.116441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.116339 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" podUID="af7cde15-0d3d-47b3-bf40-3256cfced049" containerName="payload-processing" containerID="cri-o://f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c" gracePeriod=25 Nov 01 01:09:06.228954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.228832 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" podUID="e535ab08-281e-4d91-b7fa-2647a4f8c7e8" containerName="payload-pre-processing" containerID="cri-o://76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1" gracePeriod=25 Nov 01 01:09:06.417602 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.417407 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:06.487109 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.487077 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:09:06.537334 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.536224 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7cde15-0d3d-47b3-bf40-3256cfced049-kube-api-access-85rtd\" (UniqueName: \"kubernetes.io/projected/af7cde15-0d3d-47b3-bf40-3256cfced049-kube-api-access-85rtd\") pod \"af7cde15-0d3d-47b3-bf40-3256cfced049\" (UID: \"af7cde15-0d3d-47b3-bf40-3256cfced049\") " Nov 01 01:09:06.537334 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.536540 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af7cde15-0d3d-47b3-bf40-3256cfced049-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7cde15-0d3d-47b3-bf40-3256cfced049-plugins-config-volume\") pod \"af7cde15-0d3d-47b3-bf40-3256cfced049\" (UID: \"af7cde15-0d3d-47b3-bf40-3256cfced049\") " Nov 01 01:09:06.537334 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.537238 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7cde15-0d3d-47b3-bf40-3256cfced049-plugins-config-volume" pod "af7cde15-0d3d-47b3-bf40-3256cfced049" (UID: "af7cde15-0d3d-47b3-bf40-3256cfced049"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:06.539928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.539394 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7cde15-0d3d-47b3-bf40-3256cfced049-kube-api-access-85rtd" pod "af7cde15-0d3d-47b3-bf40-3256cfced049" (UID: "af7cde15-0d3d-47b3-bf40-3256cfced049"). InnerVolumeSpecName "kube-api-access-85rtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:06.591464 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.591433 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:09:06.639411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.638426 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-kube-api-access-22tgk\" (UniqueName: \"kubernetes.io/projected/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-kube-api-access-22tgk\") pod \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\" (UID: \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\") " Nov 01 01:09:06.639411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.638553 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-plugins-config-volume\") pod \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\" (UID: \"e535ab08-281e-4d91-b7fa-2647a4f8c7e8\") " Nov 01 01:09:06.639411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.639060 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7cde15-0d3d-47b3-bf40-3256cfced049-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:06.639411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.639089 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rtd\" (UniqueName: \"kubernetes.io/projected/af7cde15-0d3d-47b3-bf40-3256cfced049-kube-api-access-85rtd\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:06.639989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.639957 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-plugins-config-volume" pod "e535ab08-281e-4d91-b7fa-2647a4f8c7e8" (UID: "e535ab08-281e-4d91-b7fa-2647a4f8c7e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:06.642151 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.642123 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-kube-api-access-22tgk" pod "e535ab08-281e-4d91-b7fa-2647a4f8c7e8" (UID: "e535ab08-281e-4d91-b7fa-2647a4f8c7e8"). InnerVolumeSpecName "kube-api-access-22tgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:06.674567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.673252 6036 generic.go:356] "Generic (PLEG): container finished" podID="e535ab08-281e-4d91-b7fa-2647a4f8c7e8" containerID="76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1" exitCode=0 Nov 01 01:09:06.674567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.673391 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" event={"ID":"e535ab08-281e-4d91-b7fa-2647a4f8c7e8","Type":"ContainerDied","Data":"76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1"} Nov 01 01:09:06.674567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.673433 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" event={"ID":"e535ab08-281e-4d91-b7fa-2647a4f8c7e8","Type":"ContainerDied","Data":"c17837963341d615aa67281d03c132abddac42d6377f36532df7582c4320512d"} Nov 01 01:09:06.674567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.673453 6036 scope.go:122] "RemoveContainer" containerID="76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1" Nov 01 01:09:06.675062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.674941 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b" Nov 01 01:09:06.677553 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.677518 6036 generic.go:356] "Generic (PLEG): container finished" podID="af7cde15-0d3d-47b3-bf40-3256cfced049" containerID="f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c" exitCode=0 Nov 01 01:09:06.677707 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.677689 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" event={"ID":"af7cde15-0d3d-47b3-bf40-3256cfced049","Type":"ContainerDied","Data":"f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c"} Nov 01 01:09:06.677779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.677735 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" event={"ID":"af7cde15-0d3d-47b3-bf40-3256cfced049","Type":"ContainerDied","Data":"70ad2ca86f07115ac35c2da7bba98add60a4fb1e87e2b85aa53ff6aebc35ea61"} Nov 01 01:09:06.677840 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.677810 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r" Nov 01 01:09:06.742468 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.742409 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tgk\" (UniqueName: \"kubernetes.io/projected/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-kube-api-access-22tgk\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:06.742732 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.742439 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e535ab08-281e-4d91-b7fa-2647a4f8c7e8-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:06.796456 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.796434 6036 scope.go:122] "RemoveContainer" containerID="76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1" Nov 01 01:09:06.804102 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:06.797981 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1\": container with ID starting with 76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1 not found: ID does not exist" containerID="76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1" Nov 01 01:09:06.804102 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.798028 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1"} err="failed to get container status \"76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1\": rpc error: code = NotFound desc = could not find container \"76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1\": container with ID starting with 76447c0b0633af0ea4a6e90d07fac4c4a2869738b7352d200f53ec88d5e174d1 not found: ID does not exist" Nov 01 01:09:06.804102 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.798056 6036 scope.go:122] "RemoveContainer" containerID="f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c" Nov 01 01:09:06.808540 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.808507 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 01:09:06.808713 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.808558 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 01:09:06.811537 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.811510 6036 scope.go:122] "RemoveContainer" containerID="f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c" Nov 01 01:09:06.811926 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:06.811906 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c\": container with ID starting with f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c not found: ID does not exist" containerID="f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c" Nov 01 01:09:06.812029 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.811935 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c"} err="failed to get container status \"f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c\": rpc error: code = NotFound desc = could not find container \"f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c\": container with ID starting with f966890c86d6632cd4dea1ce5e7d0c83fe18c64b23ce2040fb4a6f672c6c599c not found: ID does not exist" Nov 01 01:09:06.986950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.985042 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:09:06.993443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.990718 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r"] Nov 01 01:09:06.999316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:06.994544 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-603e8c-67895b6dcb-r5j6r"] Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.073547 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-credential-socket\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.073626 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-socket\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.073682 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-data\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.073738 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-podinfo\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.073833 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-envoy\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.073871 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istiod-ca-cert\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.073912 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-kube-api-access-m6bx2\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-kube-api-access-m6bx2\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.073953 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-token\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-token\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.074313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.074138 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-certs\") pod \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\" (UID: \"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61\") " Nov 01 01:09:07.075275 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.075114 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-certs" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.078077 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.077969 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-credential-socket" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.078077 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.078047 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-data" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.082594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.079595 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-socket" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.082594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.080990 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istiod-ca-cert" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:07.082886 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.082861 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-envoy" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.083700 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.083634 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-podinfo" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:07.084860 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.084828 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-token" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.085226 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.085199 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-kube-api-access-m6bx2" pod "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" (UID: "fcc1ed2c-782d-4cd8-a154-4fd877b6ef61"). InnerVolumeSpecName "kube-api-access-m6bx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.093056 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.093023 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:09:07.095823 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.095735 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:09:07.116673 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.116532 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b"] Nov 01 01:09:07.120966 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.120938 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-603e8c-55c77b8cff-dlx8b"] Nov 01 01:09:07.176045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176000 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-token\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-token\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.176260 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176073 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istiod-ca-cert\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.176260 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176126 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-envoy\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.176260 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176173 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d426904-92e2-427c-b318-2a841fec00e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d426904-92e2-427c-b318-2a841fec00e7-istio-podinfo\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176260 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176208 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-podinfo\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.176260 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176248 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-certs\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.176494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176341 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-socket\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176361 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-kube-api-access-wjktw\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-kube-api-access-wjktw\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.176494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176439 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-socket\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.176494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176462 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-kube-api-access-b84sd\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-kube-api-access-b84sd\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176525 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-certs\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176551 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-istio-token\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-istio-token\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176569 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-credential-socket\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176608 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d426904-92e2-427c-b318-2a841fec00e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d426904-92e2-427c-b318-2a841fec00e7-istiod-ca-cert\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176634 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-envoy\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176797 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176651 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-data\") pod \"3d426904-92e2-427c-b318-2a841fec00e7\" (UID: \"3d426904-92e2-427c-b318-2a841fec00e7\") " Nov 01 01:09:07.176797 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176689 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-data\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.176797 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.176733 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-credential-socket\") pod \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\" (UID: \"41c7cb30-f378-47ee-8b85-3a431de3b7e8\") " Nov 01 01:09:07.177328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177256 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177302 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177318 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177331 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177348 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177362 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177359 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-socket" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.177660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177377 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177419 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177464 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6bx2\" (UniqueName: \"kubernetes.io/projected/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61-kube-api-access-m6bx2\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.177660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177503 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-credential-socket" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.177660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177573 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-certs" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.178010 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177671 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-credential-socket" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.178010 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177775 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istiod-ca-cert" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:07.178010 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.177872 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-data" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.178325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.178075 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-data" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.178325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.178151 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d426904-92e2-427c-b318-2a841fec00e7-istiod-ca-cert" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:07.178325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.178264 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-socket" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.178533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.178492 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-certs" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.180419 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.180387 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-token" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.180670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.180648 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-istio-token" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.180778 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.180699 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-envoy" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.180778 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.180725 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-kube-api-access-b84sd" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "kube-api-access-b84sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.180997 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.180965 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d426904-92e2-427c-b318-2a841fec00e7-istio-podinfo" pod "3d426904-92e2-427c-b318-2a841fec00e7" (UID: "3d426904-92e2-427c-b318-2a841fec00e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:07.181089 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.181071 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-podinfo" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:07.181380 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.181363 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-envoy" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.182600 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.182582 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-kube-api-access-wjktw" pod "41c7cb30-f378-47ee-8b85-3a431de3b7e8" (UID: "41c7cb30-f378-47ee-8b85-3a431de3b7e8"). InnerVolumeSpecName "kube-api-access-wjktw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278676 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278710 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278727 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d426904-92e2-427c-b318-2a841fec00e7-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278741 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278758 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278772 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278786 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjktw\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-kube-api-access-wjktw\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278801 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278814 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b84sd\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-kube-api-access-b84sd\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278828 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278841 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d426904-92e2-427c-b318-2a841fec00e7-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278855 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278867 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d426904-92e2-427c-b318-2a841fec00e7-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278880 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278893 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d426904-92e2-427c-b318-2a841fec00e7-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278905 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278920 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c7cb30-f378-47ee-8b85-3a431de3b7e8-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.279048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.278933 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c7cb30-f378-47ee-8b85-3a431de3b7e8-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.284859 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.284821 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af7cde15-0d3d-47b3-bf40-3256cfced049" path="/var/lib/kubelet/pods/af7cde15-0d3d-47b3-bf40-3256cfced049/volumes" Nov 01 01:09:07.285324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.285308 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e535ab08-281e-4d91-b7fa-2647a4f8c7e8" path="/var/lib/kubelet/pods/e535ab08-281e-4d91-b7fa-2647a4f8c7e8/volumes" Nov 01 01:09:07.687084 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.687034 6036 generic.go:356] "Generic (PLEG): container finished" podID="3d426904-92e2-427c-b318-2a841fec00e7" containerID="1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c" exitCode=0 Nov 01 01:09:07.687330 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.687094 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" Nov 01 01:09:07.687330 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.687104 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" event={"ID":"3d426904-92e2-427c-b318-2a841fec00e7","Type":"ContainerDied","Data":"1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c"} Nov 01 01:09:07.687330 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.687166 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd" event={"ID":"3d426904-92e2-427c-b318-2a841fec00e7","Type":"ContainerDied","Data":"a2b4f5c081605967688fd7b47c80ed444fe937cb2ea8286ceacd2f130b7cb299"} Nov 01 01:09:07.687330 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.687189 6036 scope.go:122] "RemoveContainer" containerID="1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c" Nov 01 01:09:07.688702 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.688674 6036 generic.go:356] "Generic (PLEG): container finished" podID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerID="b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af" exitCode=0 Nov 01 01:09:07.688811 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.688735 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" Nov 01 01:09:07.688811 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.688762 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" event={"ID":"41c7cb30-f378-47ee-8b85-3a431de3b7e8","Type":"ContainerDied","Data":"b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af"} Nov 01 01:09:07.689147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.689125 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4" event={"ID":"41c7cb30-f378-47ee-8b85-3a431de3b7e8","Type":"ContainerDied","Data":"6a318b281a1e83a07fe9a958e5fd7e3f7225c36ca9d6f5b09263ea0baf297598"} Nov 01 01:09:07.691669 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.691644 6036 generic.go:356] "Generic (PLEG): container finished" podID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerID="a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93" exitCode=0 Nov 01 01:09:07.691796 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.691693 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" Nov 01 01:09:07.691796 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.691747 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" event={"ID":"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61","Type":"ContainerDied","Data":"a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93"} Nov 01 01:09:07.691910 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.691797 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g" event={"ID":"fcc1ed2c-782d-4cd8-a154-4fd877b6ef61","Type":"ContainerDied","Data":"969434da28ab519cba759d1239bcb0df2f0c86345c41e41ed06dd3f165e054fe"} Nov 01 01:09:07.699642 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.699621 6036 scope.go:122] "RemoveContainer" containerID="1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c" Nov 01 01:09:07.700034 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:07.699953 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c\": container with ID starting with 1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c not found: ID does not exist" containerID="1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c" Nov 01 01:09:07.700034 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.699994 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c"} err="failed to get container status \"1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c\": rpc error: code = NotFound desc = could not find container \"1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c\": container with ID starting with 1c0bba533613d0c0f5b654536c82ede70f3a663c2d6fb68f43979d0ae223350c not found: ID does not exist" Nov 01 01:09:07.700034 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.700019 6036 scope.go:122] "RemoveContainer" containerID="b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af" Nov 01 01:09:07.712519 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.712489 6036 scope.go:122] "RemoveContainer" containerID="b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af" Nov 01 01:09:07.712906 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:07.712885 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af\": container with ID starting with b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af not found: ID does not exist" containerID="b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af" Nov 01 01:09:07.712989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.712919 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af"} err="failed to get container status \"b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af\": rpc error: code = NotFound desc = could not find container \"b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af\": container with ID starting with b1956499aa2b5bdfd2ea69f2d5484678a554ad60ad6bf1628c7859a7cbf102af not found: ID does not exist" Nov 01 01:09:07.712989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.712946 6036 scope.go:122] "RemoveContainer" containerID="a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93" Nov 01 01:09:07.722635 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.722615 6036 scope.go:122] "RemoveContainer" containerID="a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93" Nov 01 01:09:07.722994 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:07.722972 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93\": container with ID starting with a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93 not found: ID does not exist" containerID="a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93" Nov 01 01:09:07.723074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.723007 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93"} err="failed to get container status \"a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93\": rpc error: code = NotFound desc = could not find container \"a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93\": container with ID starting with a11dfdca10639aac8ac3e918d1b69acefb54a94b2ada4a5e4ff25009da0d3a93 not found: ID does not exist" Nov 01 01:09:07.786117 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.786071 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc"] Nov 01 01:09:07.786393 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.786344 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" podUID="7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" containerName="maas-api" containerID="cri-o://61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027" gracePeriod=30 Nov 01 01:09:07.805535 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.805494 6036 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 01:09:07.805725 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.805562 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 01:09:07.858335 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.858256 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g"] Nov 01 01:09:07.864405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.864373 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-qx42g"] Nov 01 01:09:07.970450 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.970246 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:07.985407 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.985368 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4"] Nov 01 01:09:07.986773 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:07.986750 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-64lv4"] Nov 01 01:09:08.080073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.080048 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 01:09:08.095903 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.095870 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd"] Nov 01 01:09:08.100485 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.100458 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-h7wvd"] Nov 01 01:09:08.192859 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.192586 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls\") pod \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " Nov 01 01:09:08.192859 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.192696 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-kube-api-access-whgcr\" (UniqueName: \"kubernetes.io/projected/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-kube-api-access-whgcr\") pod \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " Nov 01 01:09:08.193477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.192871 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-metrics-certs\") pod \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " Nov 01 01:09:08.193477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.192942 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-openshift-service-ca\") pod \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\" (UID: \"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4\") " Nov 01 01:09:08.199191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.196798 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-kube-api-access-whgcr" pod "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" (UID: "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4"). InnerVolumeSpecName "kube-api-access-whgcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:08.233358 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.202071 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-metrics-certs" pod "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" (UID: "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:09:08.233358 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.216518 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-openshift-service-ca" pod "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" (UID: "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:08.239308 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.236806 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls" pod "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" (UID: "7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:09:08.294871 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.294832 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:08.294871 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.294864 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whgcr\" (UniqueName: \"kubernetes.io/projected/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-kube-api-access-whgcr\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:08.294871 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.294876 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:08.295128 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.294887 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:08.705660 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.705616 6036 generic.go:356] "Generic (PLEG): container finished" podID="7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" containerID="61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027" exitCode=0 Nov 01 01:09:08.705854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.705749 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" Nov 01 01:09:08.705854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.705755 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" event={"ID":"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4","Type":"ContainerDied","Data":"61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027"} Nov 01 01:09:08.705854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.705846 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc" event={"ID":"7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4","Type":"ContainerDied","Data":"e49a3fde4f9c6183b90db13cd02577f1b1434b869fc933254e53236cea3bad3b"} Nov 01 01:09:08.706043 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.705875 6036 scope.go:122] "RemoveContainer" containerID="61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027" Nov 01 01:09:08.724850 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.724819 6036 scope.go:122] "RemoveContainer" containerID="61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027" Nov 01 01:09:08.725271 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:08.725248 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027\": container with ID starting with 61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027 not found: ID does not exist" containerID="61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027" Nov 01 01:09:08.725369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.725279 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027"} err="failed to get container status \"61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027\": rpc error: code = NotFound desc = could not find container \"61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027\": container with ID starting with 61de36f6f0e959eb695c4368f3e3a0ffd54c5112c603af81a5f3d4dce6781027 not found: ID does not exist" Nov 01 01:09:08.905102 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.905060 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc"] Nov 01 01:09:08.908649 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.908605 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-32439f-7cd45856bb-h7kxc"] Nov 01 01:09:08.964575 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.964482 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:08.965009 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.964990 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:08.965902 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:08.965879 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:09.251810 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:09.251723 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d426904-92e2-427c-b318-2a841fec00e7" path="/var/lib/kubelet/pods/3d426904-92e2-427c-b318-2a841fec00e7/volumes" Nov 01 01:09:09.252487 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:09.252467 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" path="/var/lib/kubelet/pods/41c7cb30-f378-47ee-8b85-3a431de3b7e8/volumes" Nov 01 01:09:09.252975 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:09.252952 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" path="/var/lib/kubelet/pods/7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4/volumes" Nov 01 01:09:09.253803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:09.253782 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" path="/var/lib/kubelet/pods/fcc1ed2c-782d-4cd8-a154-4fd877b6ef61/volumes" Nov 01 01:09:09.634677 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:09.634649 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:09.841177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:09.841140 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:10.573338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:10.573301 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:15.127843 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:15.127806 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:16.871596 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:16.871549 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:17.170074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:17.169984 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:17.170604 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:17.170576 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2"] Nov 01 01:09:17.170823 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:17.170799 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" containerID="cri-o://22159a08fa8e038360b70f720ee88459c65596a294cee53f156cc1e39bbe59ce" gracePeriod=30 Nov 01 01:09:17.202457 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:17.202427 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr"] Nov 01 01:09:17.202645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:17.202625 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" containerID="cri-o://5038517ecb36ade046090024bc922ba373a71f292552ffcc694112908b14c55e" gracePeriod=30 Nov 01 01:09:17.407327 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:17.407246 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:18.764521 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:18.764481 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr"] Nov 01 01:09:18.765019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:18.764752 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" containerID="cri-o://504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e" gracePeriod=30 Nov 01 01:09:19.522389 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:19.522345 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:09:19.958803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.958753 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz"] Nov 01 01:09:19.959711 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959690 6036 cpu_manager.go:414] "containerMap: removing container" podUID="7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" containerName="maas-api" Nov 01 01:09:19.959790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959715 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" containerName="maas-api" Nov 01 01:09:19.959790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959732 6036 cpu_manager.go:414] "containerMap: removing container" podUID="af7cde15-0d3d-47b3-bf40-3256cfced049" containerName="payload-processing" Nov 01 01:09:19.959790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959740 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7cde15-0d3d-47b3-bf40-3256cfced049" containerName="payload-processing" Nov 01 01:09:19.959790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959761 6036 cpu_manager.go:414] "containerMap: removing container" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" Nov 01 01:09:19.959790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959770 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" Nov 01 01:09:19.959790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959790 6036 cpu_manager.go:414] "containerMap: removing container" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" Nov 01 01:09:19.959985 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959797 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" Nov 01 01:09:19.959985 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959808 6036 cpu_manager.go:414] "containerMap: removing container" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" Nov 01 01:09:19.959985 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959815 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" Nov 01 01:09:19.959985 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959825 6036 cpu_manager.go:414] "containerMap: removing container" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" Nov 01 01:09:19.959985 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959832 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" Nov 01 01:09:19.959985 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959869 6036 cpu_manager.go:414] "containerMap: removing container" podUID="e535ab08-281e-4d91-b7fa-2647a4f8c7e8" containerName="payload-pre-processing" Nov 01 01:09:19.959985 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.959877 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e535ab08-281e-4d91-b7fa-2647a4f8c7e8" containerName="payload-pre-processing" Nov 01 01:09:19.960220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.960034 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1c4e42-1f20-42b8-8144-6bb8f3c0c1c4" containerName="maas-api" Nov 01 01:09:19.960220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.960045 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7cde15-0d3d-47b3-bf40-3256cfced049" containerName="payload-processing" Nov 01 01:09:19.960220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.960060 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d426904-92e2-427c-b318-2a841fec00e7" containerName="istio-proxy" Nov 01 01:09:19.960220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.960070 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c7cb30-f378-47ee-8b85-3a431de3b7e8" containerName="istio-proxy" Nov 01 01:09:19.960220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.960081 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc1ed2c-782d-4cd8-a154-4fd877b6ef61" containerName="istio-proxy" Nov 01 01:09:19.960220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.960091 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4a2c30-4b8f-4d6e-90f8-8569f072d908" containerName="istio-proxy" Nov 01 01:09:19.960220 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.960101 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e535ab08-281e-4d91-b7fa-2647a4f8c7e8" containerName="payload-pre-processing" Nov 01 01:09:19.965954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:19.965921 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.016544 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.016512 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz"] Nov 01 01:09:20.063725 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.063697 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1402e8-serving-cert\"" Nov 01 01:09:20.145625 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.145590 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpsqx\" (UniqueName: \"kubernetes.io/projected/50a18b44-be40-48e3-b4f3-383170ca5c7a-kube-api-access-kpsqx\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.145625 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.145628 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a18b44-be40-48e3-b4f3-383170ca5c7a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.145842 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.145756 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-metrics-certs\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.145880 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.145838 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.229788 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.229706 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:20.246774 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.246744 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpsqx\" (UniqueName: \"kubernetes.io/projected/50a18b44-be40-48e3-b4f3-383170ca5c7a-kube-api-access-kpsqx\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.246979 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.246789 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a18b44-be40-48e3-b4f3-383170ca5c7a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.246979 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.246855 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-metrics-certs\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.246979 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.246955 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.247192 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:20.247117 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-serving-cert: secret "maas-api-e2e-api-b-1402e8-serving-cert" not found Nov 01 01:09:20.247192 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:20.247185 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls podName:50a18b44-be40-48e3-b4f3-383170ca5c7a nodeName:}" failed. No retries permitted until 2026-11-01 01:09:20.747164965 +0000 UTC m=+2705.622385962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls") pod "maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" (UID: "50a18b44-be40-48e3-b4f3-383170ca5c7a") : secret "maas-api-e2e-api-b-1402e8-serving-cert" not found Nov 01 01:09:20.247498 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.247479 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a18b44-be40-48e3-b4f3-383170ca5c7a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.249508 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.249487 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-metrics-certs\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.313875 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.313844 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpsqx\" (UniqueName: \"kubernetes.io/projected/50a18b44-be40-48e3-b4f3-383170ca5c7a-kube-api-access-kpsqx\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.504047 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.503970 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:20.710558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.710528 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:20.753210 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.753179 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.755883 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.755820 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls\") pod \"maas-api-e2e-api-b-1402e8-6f798896b8-x77hz\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:20.880355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:20.880323 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:21.272643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:21.272611 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz"] Nov 01 01:09:21.806077 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:21.806044 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" event={"ID":"50a18b44-be40-48e3-b4f3-383170ca5c7a","Type":"ContainerStarted","Data":"bf9f2e8642708d51de77cd3019ab1135615ecb9b470b9dd389416d5c3a311f27"} Nov 01 01:09:22.842142 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:22.842107 6036 generic.go:356] "Generic (PLEG): container finished" podID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerID="5038517ecb36ade046090024bc922ba373a71f292552ffcc694112908b14c55e" exitCode=0 Nov 01 01:09:22.842758 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:22.842735 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" event={"ID":"0adc53d6-54da-4b31-8a11-c59e6216b215","Type":"ContainerDied","Data":"5038517ecb36ade046090024bc922ba373a71f292552ffcc694112908b14c55e"} Nov 01 01:09:22.854534 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:22.853803 6036 generic.go:356] "Generic (PLEG): container finished" podID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerID="22159a08fa8e038360b70f720ee88459c65596a294cee53f156cc1e39bbe59ce" exitCode=0 Nov 01 01:09:22.854534 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:22.853997 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" event={"ID":"cb59fb86-48cd-448e-9d57-66f80ee699c6","Type":"ContainerDied","Data":"22159a08fa8e038360b70f720ee88459c65596a294cee53f156cc1e39bbe59ce"} Nov 01 01:09:22.856633 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:22.856606 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" event={"ID":"50a18b44-be40-48e3-b4f3-383170ca5c7a","Type":"ContainerStarted","Data":"58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd"} Nov 01 01:09:22.857935 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:22.857913 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:23.033081 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.033008 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" podStartSLOduration=3.187834545 podStartE2EDuration="4.032983669s" totalImagesPullingTime="845.149124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:09:23.017051722 +0000 UTC m=+2707.892272739" watchObservedRunningTime="2026-11-01 01:09:23.032983669 +0000 UTC m=+2707.908204687" Nov 01 01:09:23.044208 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.044178 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 01:09:23.139738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.139711 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:23.143204 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143178 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-credential-socket\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.143375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143224 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-podinfo\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.143375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143268 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-token\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-token\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.143375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143305 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-socket\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.143533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143512 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-envoy\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.143597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143514 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-credential-socket" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.143597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143566 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-socket" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.143597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143555 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-socket\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.143768 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143635 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-credential-socket\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.143768 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143680 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-data\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.143768 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143731 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-podinfo\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.143924 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143786 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-kube-api-access-crj9b\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-kube-api-access-crj9b\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.143924 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143818 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb59fb86-48cd-448e-9d57-66f80ee699c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb59fb86-48cd-448e-9d57-66f80ee699c6-istiod-ca-cert\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.143924 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143877 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-data\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.144075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143967 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-token\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-token\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.144075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.143999 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-kube-api-access-bjdxr\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-kube-api-access-bjdxr\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.144075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144004 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-data" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.144075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144032 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-credential-socket" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.144481 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144057 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-certs\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.144481 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144086 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-socket" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.144481 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144129 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-envoy\") pod \"cb59fb86-48cd-448e-9d57-66f80ee699c6\" (UID: \"cb59fb86-48cd-448e-9d57-66f80ee699c6\") " Nov 01 01:09:23.144481 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144186 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0adc53d6-54da-4b31-8a11-c59e6216b215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc53d6-54da-4b31-8a11-c59e6216b215-istiod-ca-cert\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.144481 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144226 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-certs" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.144481 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144424 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-data" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.144802 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144609 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb59fb86-48cd-448e-9d57-66f80ee699c6-istiod-ca-cert" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:23.144946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144928 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.145020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144954 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.145020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144967 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.145020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144982 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb59fb86-48cd-448e-9d57-66f80ee699c6-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.145020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144991 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.145020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.144999 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.145020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.145007 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.145020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.145015 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.145442 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.145305 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adc53d6-54da-4b31-8a11-c59e6216b215-istiod-ca-cert" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:23.256919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.255786 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-certs\") pod \"0adc53d6-54da-4b31-8a11-c59e6216b215\" (UID: \"0adc53d6-54da-4b31-8a11-c59e6216b215\") " Nov 01 01:09:23.256919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.256626 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc53d6-54da-4b31-8a11-c59e6216b215-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.257381 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.257335 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-certs" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.270017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.269951 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-token" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:23.311608 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.311550 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-token" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:23.324331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.322486 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-kube-api-access-bjdxr" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "kube-api-access-bjdxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:23.330578 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.330513 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-podinfo" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:23.332529 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.332492 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-podinfo" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:23.333612 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.333571 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-envoy" pod "0adc53d6-54da-4b31-8a11-c59e6216b215" (UID: "0adc53d6-54da-4b31-8a11-c59e6216b215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.336843 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.336813 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-kube-api-access-crj9b" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "kube-api-access-crj9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:23.338900 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.338850 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-envoy" pod "cb59fb86-48cd-448e-9d57-66f80ee699c6" (UID: "cb59fb86-48cd-448e-9d57-66f80ee699c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361386 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361425 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjdxr\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-kube-api-access-bjdxr\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361442 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361457 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361471 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361485 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361499 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc53d6-54da-4b31-8a11-c59e6216b215-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361513 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb59fb86-48cd-448e-9d57-66f80ee699c6-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.401323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.361527 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crj9b\" (UniqueName: \"kubernetes.io/projected/cb59fb86-48cd-448e-9d57-66f80ee699c6-kube-api-access-crj9b\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.796882 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.796824 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h6nmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": read tcp 10.134.0.2:52928->10.134.0.90:15021: read: connection reset by peer" start-of-body= Nov 01 01:09:23.797039 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.796923 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": read tcp 10.134.0.2:52928->10.134.0.90:15021: read: connection reset by peer" Nov 01 01:09:23.872325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.867867 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" Nov 01 01:09:23.872325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.867923 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr" event={"ID":"0adc53d6-54da-4b31-8a11-c59e6216b215","Type":"ContainerDied","Data":"9e711aa21b83af0412ad574f052371be1311e499ea9e0eec7253cb1aa661d468"} Nov 01 01:09:23.872325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.868001 6036 scope.go:122] "RemoveContainer" containerID="5038517ecb36ade046090024bc922ba373a71f292552ffcc694112908b14c55e" Nov 01 01:09:23.877368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.877346 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" Nov 01 01:09:23.878099 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.878075 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2" event={"ID":"cb59fb86-48cd-448e-9d57-66f80ee699c6","Type":"ContainerDied","Data":"bd07df52d56645bc6a34989e176b3be533f65874fbf37e17fbd2467f552fc289"} Nov 01 01:09:23.897224 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:23.897104 6036 scope.go:122] "RemoveContainer" containerID="22159a08fa8e038360b70f720ee88459c65596a294cee53f156cc1e39bbe59ce" Nov 01 01:09:24.038901 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.038873 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr"] Nov 01 01:09:24.047446 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.047377 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 01:09:24.089337 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.089297 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8crtr"] Nov 01 01:09:24.110461 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110425 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istiod-ca-cert\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110631 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110503 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-data\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110631 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110537 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-podinfo\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110631 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110563 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-envoy\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110769 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110632 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-socket\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110769 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110659 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-kube-api-access-4l52s\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-kube-api-access-4l52s\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110769 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110743 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-certs\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110769 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-token\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-token\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110832 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-credential-socket\") pod \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\" (UID: \"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2\") " Nov 01 01:09:24.110921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110858 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istiod-ca-cert" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:24.110921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.110866 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-data" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:24.111259 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.111166 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-certs" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:24.111259 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.111214 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-socket" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:24.111646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.111396 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.111646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.111414 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.111646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.111428 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.111646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.111441 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.111646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.111411 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-credential-socket" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:24.113163 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.113137 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-envoy" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:24.113527 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.113509 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-kube-api-access-4l52s" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "kube-api-access-4l52s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:24.113612 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.113574 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-podinfo" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:24.113612 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.113582 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-token" pod "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" (UID: "5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:24.203927 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.203889 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2"] Nov 01 01:09:24.206019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.205995 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-8flh2"] Nov 01 01:09:24.212446 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.212426 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l52s\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-kube-api-access-4l52s\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.212539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.212450 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.212539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.212460 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.212539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.212469 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.212539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.212478 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:24.883753 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.883722 6036 generic.go:356] "Generic (PLEG): container finished" podID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerID="504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e" exitCode=0 Nov 01 01:09:24.884207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.883780 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" Nov 01 01:09:24.884207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.883806 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" event={"ID":"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2","Type":"ContainerDied","Data":"504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e"} Nov 01 01:09:24.884207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.883888 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr" event={"ID":"5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2","Type":"ContainerDied","Data":"acd84377844dfd3cb6e97352a9e3d27b01392db4a156c7d7aa59704660f0d328"} Nov 01 01:09:24.884207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.883912 6036 scope.go:122] "RemoveContainer" containerID="504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e" Nov 01 01:09:24.894845 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.894826 6036 scope.go:122] "RemoveContainer" containerID="504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e" Nov 01 01:09:24.895106 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:24.895085 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e\": container with ID starting with 504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e not found: ID does not exist" containerID="504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e" Nov 01 01:09:24.895164 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:24.895119 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e"} err="failed to get container status \"504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e\": rpc error: code = NotFound desc = could not find container \"504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e\": container with ID starting with 504ce7f074170eb288f93fcdcc079301808d2ab53e8baa78896287d99c35c97e not found: ID does not exist" Nov 01 01:09:25.051633 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:25.051595 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr"] Nov 01 01:09:25.102548 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:25.102517 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h6nmr"] Nov 01 01:09:25.250383 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:25.250304 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" path="/var/lib/kubelet/pods/0adc53d6-54da-4b31-8a11-c59e6216b215/volumes" Nov 01 01:09:25.250783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:25.250769 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" path="/var/lib/kubelet/pods/5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2/volumes" Nov 01 01:09:25.251179 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:25.251166 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" path="/var/lib/kubelet/pods/cb59fb86-48cd-448e-9d57-66f80ee699c6/volumes" Nov 01 01:09:25.477425 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:25.477386 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:29.892164 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:29.892136 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:09:30.438111 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:30.438078 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:31.197256 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:31.197216 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:35.773079 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:35.773036 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:36.026171 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:36.026063 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:40.853809 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:40.853766 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:45.513439 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:45.513403 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:45.727606 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:45.727573 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:45.936831 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:45.936796 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:49.088981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.088949 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb"] Nov 01 01:09:49.089526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089501 6036 cpu_manager.go:414] "containerMap: removing container" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" Nov 01 01:09:49.089526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089513 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" Nov 01 01:09:49.089616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089534 6036 cpu_manager.go:414] "containerMap: removing container" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" Nov 01 01:09:49.089616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089539 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" Nov 01 01:09:49.089616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089547 6036 cpu_manager.go:414] "containerMap: removing container" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" Nov 01 01:09:49.089616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089551 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" Nov 01 01:09:49.089761 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089656 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdd8a31-13ec-4bdf-aa71-b6d815af2dd2" containerName="istio-proxy" Nov 01 01:09:49.089761 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089665 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb59fb86-48cd-448e-9d57-66f80ee699c6" containerName="istio-proxy" Nov 01 01:09:49.089761 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.089673 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adc53d6-54da-4b31-8a11-c59e6216b215" containerName="istio-proxy" Nov 01 01:09:49.093047 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.093031 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130385 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130344 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130391 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde2721d-f699-4371-b0e8-584978a0df59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130467 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130509 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130528 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kbw\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-kube-api-access-62kbw\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130733 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130655 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130733 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130696 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130733 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130713 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde2721d-f699-4371-b0e8-584978a0df59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.130874 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.130858 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.215390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.214829 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb"] Nov 01 01:09:49.231873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.231834 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.231952 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.231984 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde2721d-f699-4371-b0e8-584978a0df59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232011 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232044 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232074 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kbw\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-kube-api-access-62kbw\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232169 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232228 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232252 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde2721d-f699-4371-b0e8-584978a0df59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232546 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232321 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232546 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232344 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232753 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232726 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.232854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.232797 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.233031 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.233009 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde2721d-f699-4371-b0e8-584978a0df59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.234819 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.234792 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.235157 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.235138 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde2721d-f699-4371-b0e8-584978a0df59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.295090 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.295046 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.295278 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.295176 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kbw\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-kube-api-access-62kbw\") pod \"maas-default-gateway-openshift-default-67bf986f99-gbrgb\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.407742 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.407654 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:49.519530 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:09:49.519492 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:09:49.840689 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.840664 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb"] Nov 01 01:09:49.844470 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:09:49.844440 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde2721d_f699_4371_b0e8_584978a0df59.slice/crio-84654cd77e8aebfc13841f9c6b4bfc90c7264ef7daaee69a708a50be531a812e WatchSource:0}: Error finding container 84654cd77e8aebfc13841f9c6b4bfc90c7264ef7daaee69a708a50be531a812e: Status 404 returned error can't find the container with id 84654cd77e8aebfc13841f9c6b4bfc90c7264ef7daaee69a708a50be531a812e Nov 01 01:09:49.847154 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.847119 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:49.847244 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.847187 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:49.847244 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:49.847215 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:50.049444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:50.049407 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" event={"ID":"cde2721d-f699-4371-b0e8-584978a0df59","Type":"ContainerStarted","Data":"2d2d629ccc961d7975a684861b8827442fd31344270c73c09cea188cd66f23ea"} Nov 01 01:09:50.049653 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:50.049459 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" event={"ID":"cde2721d-f699-4371-b0e8-584978a0df59","Type":"ContainerStarted","Data":"84654cd77e8aebfc13841f9c6b4bfc90c7264ef7daaee69a708a50be531a812e"} Nov 01 01:09:50.172889 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:50.172785 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podStartSLOduration=1.172770779 podStartE2EDuration="1.172770779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:09:50.171313378 +0000 UTC m=+2735.046534392" watchObservedRunningTime="2026-11-01 01:09:50.172770779 +0000 UTC m=+2735.047991793" Nov 01 01:09:50.408349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:50.408276 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:50.410056 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:50.410030 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gbrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 01:09:50.410196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:50.410094 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 01:09:50.670342 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:50.667427 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:51.408505 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:51.408461 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gbrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 01:09:51.408920 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:51.408521 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 01:09:52.408210 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:52.408172 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gbrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 01:09:52.408438 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:52.408238 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 01:09:53.408793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:53.408755 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gbrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 01:09:53.409205 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:53.408808 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 01:09:54.408610 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:54.408575 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gbrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 01:09:54.408880 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:54.408626 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 01:09:55.408694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:55.408655 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gbrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 01:09:55.408909 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:55.408714 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 01:09:55.771104 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:55.770999 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:56.409082 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:56.409040 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gbrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 01:09:56.409533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:56.409124 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 01:09:57.408945 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:57.408903 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gbrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 01:09:57.409142 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:57.408959 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 01:09:58.413268 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:58.413239 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:58.413786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:58.413560 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:58.414327 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:58.414310 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:09:58.864984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:58.864943 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:09:59.073835 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:09:59.073807 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:00.673308 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:00.673265 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:04.251136 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.251097 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt"] Nov 01 01:10:04.255154 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.255134 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.349726 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.349689 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbkt\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-kube-api-access-tqbkt\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.350018 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.349990 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.350432 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.350409 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.350672 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.350647 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.350797 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.350727 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.350797 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.350765 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.350917 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.350808 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.351818 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.351794 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95829749-19bd-4ac0-86e8-a3635a805ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.352420 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.352400 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.362012 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.361982 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt"] Nov 01 01:10:04.453839 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.453785 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbkt\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-kube-api-access-tqbkt\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454049 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.453847 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454049 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.453872 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454049 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.453890 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454049 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.453995 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.454045 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.454094 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.454146 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95829749-19bd-4ac0-86e8-a3635a805ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.454274 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454552 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.454413 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454552 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.454504 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454744 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.454724 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.454877 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.454752 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.455029 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.455009 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95829749-19bd-4ac0-86e8-a3635a805ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.456525 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.456502 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.456739 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.456720 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.515913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.515835 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.515913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.515839 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbkt\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-kube-api-access-tqbkt\") pod \"maas-default-gateway-openshift-default-67bf986f99-swvvt\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:04.570382 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:04.570346 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:05.004663 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.004633 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt"] Nov 01 01:10:05.006720 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:10:05.006686 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95829749_19bd_4ac0_86e8_a3635a805ea7.slice/crio-1b1bcf2500b99a3c5c9e0335b3fdf8df631691dd4d2666edd59b2a287e2bf21d WatchSource:0}: Error finding container 1b1bcf2500b99a3c5c9e0335b3fdf8df631691dd4d2666edd59b2a287e2bf21d: Status 404 returned error can't find the container with id 1b1bcf2500b99a3c5c9e0335b3fdf8df631691dd4d2666edd59b2a287e2bf21d Nov 01 01:10:05.009392 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.009359 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:10:05.009490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.009424 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:10:05.009490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.009456 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:10:05.157275 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.157227 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" event={"ID":"95829749-19bd-4ac0-86e8-a3635a805ea7","Type":"ContainerStarted","Data":"0b94feb6a1f7bf9b4d1500c8bc2bb35744bff3af84c5ae182e3fb11a410f68dc"} Nov 01 01:10:05.157498 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.157371 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" event={"ID":"95829749-19bd-4ac0-86e8-a3635a805ea7","Type":"ContainerStarted","Data":"1b1bcf2500b99a3c5c9e0335b3fdf8df631691dd4d2666edd59b2a287e2bf21d"} Nov 01 01:10:05.285573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.285521 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podStartSLOduration=1.285506906 podStartE2EDuration="1.285506906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:10:05.281896888 +0000 UTC m=+2750.157117903" watchObservedRunningTime="2026-11-01 01:10:05.285506906 +0000 UTC m=+2750.160727921" Nov 01 01:10:05.573474 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.573379 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:05.673701 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:05.673657 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:06.573189 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:06.573149 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-swvvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 01:10:06.573755 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:06.573234 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 01:10:07.570864 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:07.570814 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-swvvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 01:10:07.571094 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:07.570900 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 01:10:08.571457 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:08.571426 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-swvvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 01:10:08.571953 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:08.571920 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 01:10:09.570764 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:09.570722 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-swvvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 01:10:09.571001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:09.570794 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 01:10:10.540815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:10.540779 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:10.572811 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:10.572176 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-swvvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 01:10:10.572811 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:10.572249 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 01:10:11.570965 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:11.570922 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-swvvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 01:10:11.571431 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:11.571027 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 01:10:12.571444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:12.571404 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-swvvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 01:10:12.571870 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:12.571469 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 01:10:13.626858 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:13.626822 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:13.627388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:13.627181 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:13.628171 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:13.628149 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:10:15.379709 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:15.379671 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:15.688548 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:15.688455 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:17.165059 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:17.164983 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:17.400406 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:17.400374 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:17.756444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:17.756411 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:18.297994 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:18.297964 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:19.505349 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:10:19.505315 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:10:22.878425 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:22.878386 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:24.990213 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:24.990180 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:25.260092 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:25.260011 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:25.275844 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:25.275811 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d"] Nov 01 01:10:25.276034 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:25.276000 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" containerID="cri-o://d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe" gracePeriod=30 Nov 01 01:10:25.541690 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:25.541662 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:28.187672 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:28.187638 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:30.156140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.156086 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:30.473865 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.473757 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:30.550818 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.550794 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:10:30.659959 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.659927 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-kube-api-access-2dc99\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-kube-api-access-2dc99\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660015 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-credential-socket\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660046 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-data\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660159 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-envoy\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660236 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-podinfo\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660318 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-socket\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660359 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istiod-ca-cert\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660538 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660391 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-token\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-token\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660538 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660425 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-credential-socket" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:30.660538 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660436 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-data" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:30.660538 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660476 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-certs\") pod \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\" (UID: \"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6\") " Nov 01 01:10:30.660746 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660593 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-socket" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:30.660746 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660695 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istiod-ca-cert" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:10:30.660848 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.660821 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-certs" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:30.661173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.661107 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:30.661173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.661128 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:30.661173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.661142 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:30.661173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.661154 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:30.661173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.661163 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:30.662498 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.662473 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-kube-api-access-2dc99" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "kube-api-access-2dc99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:30.662777 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.662755 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-envoy" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:30.663091 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.663064 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-podinfo" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:10:30.663161 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.663099 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-token" pod "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" (UID: "244bd1e4-d0c1-45b9-a8bc-ca172ba181a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:30.763157 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.763063 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:30.763157 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.763109 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:30.763157 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.763124 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:30.763157 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:30.763140 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dc99\" (UniqueName: \"kubernetes.io/projected/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6-kube-api-access-2dc99\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:31.332038 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.332001 6036 generic.go:356] "Generic (PLEG): container finished" podID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerID="d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe" exitCode=0 Nov 01 01:10:31.332509 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.332056 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" Nov 01 01:10:31.332509 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.332054 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" event={"ID":"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6","Type":"ContainerDied","Data":"d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe"} Nov 01 01:10:31.332509 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.332182 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d" event={"ID":"244bd1e4-d0c1-45b9-a8bc-ca172ba181a6","Type":"ContainerDied","Data":"77915e6733efd7d82462088ecfc79c12a064c937ad332f5d2c7fc0d978d8a6d3"} Nov 01 01:10:31.332509 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.332212 6036 scope.go:122] "RemoveContainer" containerID="d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe" Nov 01 01:10:31.342267 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.342245 6036 scope.go:122] "RemoveContainer" containerID="807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181" Nov 01 01:10:31.354052 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.354027 6036 scope.go:122] "RemoveContainer" containerID="d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe" Nov 01 01:10:31.354654 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:10:31.354502 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe\": container with ID starting with d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe not found: ID does not exist" containerID="d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe" Nov 01 01:10:31.354654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.354546 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe"} err="failed to get container status \"d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe\": rpc error: code = NotFound desc = could not find container \"d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe\": container with ID starting with d1ee80261dac10b9d985602b06c1928f9f8068da493d21fada8d26e5eb73edbe not found: ID does not exist" Nov 01 01:10:31.354654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.354572 6036 scope.go:122] "RemoveContainer" containerID="807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181" Nov 01 01:10:31.355786 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:10:31.354958 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181\": container with ID starting with 807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181 not found: ID does not exist" containerID="807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181" Nov 01 01:10:31.355786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.355011 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181"} err="failed to get container status \"807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181\": rpc error: code = NotFound desc = could not find container \"807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181\": container with ID starting with 807614bd30654480e69c4a195e207ffa7d0398224817baf057986ac1491fa181 not found: ID does not exist" Nov 01 01:10:31.557076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.557027 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d"] Nov 01 01:10:31.577006 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.576972 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-603e8c-openshift-default-7889fcc5d7-5lr7d"] Nov 01 01:10:31.955493 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:31.955413 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:33.251196 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:33.251158 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" path="/var/lib/kubelet/pods/244bd1e4-d0c1-45b9-a8bc-ca172ba181a6/volumes" Nov 01 01:10:33.758830 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:33.758793 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:36.517548 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:36.517519 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:36.762884 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:36.762847 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s"] Nov 01 01:10:36.985462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:36.985426 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:38.203653 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:38.203618 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:38.441248 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:38.441214 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:39.230454 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:39.230412 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:40.794032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:40.793989 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:41.014737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:41.014697 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:43.904885 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:43.904785 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:44.465536 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:44.465500 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:44.700636 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:44.700584 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:45.554822 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:45.554769 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:45.801773 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:45.801739 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:46.042681 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:46.042648 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:46.552238 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:46.552184 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:49.504862 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:10:49.504821 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:10:51.300188 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:51.300154 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:51.784700 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:51.784656 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" containerID="cri-o://062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4" gracePeriod=45 Nov 01 01:10:52.052611 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.052544 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 01:10:52.098275 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098237 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-home\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-home\") pod \"828bbae3-be6b-45df-8516-9f659098e61d\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " Nov 01 01:10:52.098487 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098461 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-home" pod "828bbae3-be6b-45df-8516-9f659098e61d" (UID: "828bbae3-be6b-45df-8516-9f659098e61d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:52.098536 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098499 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828bbae3-be6b-45df-8516-9f659098e61d-kube-api-access-7bgfm\" (UniqueName: \"kubernetes.io/projected/828bbae3-be6b-45df-8516-9f659098e61d-kube-api-access-7bgfm\") pod \"828bbae3-be6b-45df-8516-9f659098e61d\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " Nov 01 01:10:52.098536 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098524 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-model-cache\") pod \"828bbae3-be6b-45df-8516-9f659098e61d\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " Nov 01 01:10:52.098634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098548 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/828bbae3-be6b-45df-8516-9f659098e61d-tls-certs\" (UniqueName: \"kubernetes.io/secret/828bbae3-be6b-45df-8516-9f659098e61d-tls-certs\") pod \"828bbae3-be6b-45df-8516-9f659098e61d\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " Nov 01 01:10:52.098634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098578 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-dshm\") pod \"828bbae3-be6b-45df-8516-9f659098e61d\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " Nov 01 01:10:52.098634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098604 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-tmp-dir\") pod \"828bbae3-be6b-45df-8516-9f659098e61d\" (UID: \"828bbae3-be6b-45df-8516-9f659098e61d\") " Nov 01 01:10:52.098804 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098731 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-model-cache" pod "828bbae3-be6b-45df-8516-9f659098e61d" (UID: "828bbae3-be6b-45df-8516-9f659098e61d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:52.098937 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.098905 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-tmp-dir" pod "828bbae3-be6b-45df-8516-9f659098e61d" (UID: "828bbae3-be6b-45df-8516-9f659098e61d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:52.099173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.099156 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.099240 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.099178 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.099240 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.099193 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.101012 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.100985 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828bbae3-be6b-45df-8516-9f659098e61d-tls-certs" pod "828bbae3-be6b-45df-8516-9f659098e61d" (UID: "828bbae3-be6b-45df-8516-9f659098e61d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:10:52.101105 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.101008 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828bbae3-be6b-45df-8516-9f659098e61d-kube-api-access-7bgfm" pod "828bbae3-be6b-45df-8516-9f659098e61d" (UID: "828bbae3-be6b-45df-8516-9f659098e61d"). InnerVolumeSpecName "kube-api-access-7bgfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:52.101105 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.100987 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-dshm" pod "828bbae3-be6b-45df-8516-9f659098e61d" (UID: "828bbae3-be6b-45df-8516-9f659098e61d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:52.200556 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.200515 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bgfm\" (UniqueName: \"kubernetes.io/projected/828bbae3-be6b-45df-8516-9f659098e61d-kube-api-access-7bgfm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.200556 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.200547 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828bbae3-be6b-45df-8516-9f659098e61d-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.200556 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.200560 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828bbae3-be6b-45df-8516-9f659098e61d-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.482338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.482227 6036 generic.go:356] "Generic (PLEG): container finished" podID="828bbae3-be6b-45df-8516-9f659098e61d" containerID="062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4" exitCode=0 Nov 01 01:10:52.482338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.482271 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" Nov 01 01:10:52.482338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.482312 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" event={"ID":"828bbae3-be6b-45df-8516-9f659098e61d","Type":"ContainerDied","Data":"062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4"} Nov 01 01:10:52.482906 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.482357 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s" event={"ID":"828bbae3-be6b-45df-8516-9f659098e61d","Type":"ContainerDied","Data":"3133d1193a513ac5c536d64b2e6d841b060f5b2fce6bbc00c27014a0c6485db5"} Nov 01 01:10:52.482906 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.482373 6036 scope.go:122] "RemoveContainer" containerID="062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4" Nov 01 01:10:52.493037 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.493021 6036 scope.go:122] "RemoveContainer" containerID="062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4" Nov 01 01:10:52.493326 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:10:52.493278 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4\": container with ID starting with 062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4 not found: ID does not exist" containerID="062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4" Nov 01 01:10:52.493422 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.493331 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4"} err="failed to get container status \"062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4\": rpc error: code = NotFound desc = could not find container \"062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4\": container with ID starting with 062901cad2597b67e71945fd7da923ba132ea96758f2ffcc98389e7d4d536bb4 not found: ID does not exist" Nov 01 01:10:52.657023 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.656986 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s"] Nov 01 01:10:52.668380 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:52.668346 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/distinct-2-w3-5a9c19-kserve-d7896b869-nzb8s"] Nov 01 01:10:53.255675 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:53.255628 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828bbae3-be6b-45df-8516-9f659098e61d" path="/var/lib/kubelet/pods/828bbae3-be6b-45df-8516-9f659098e61d/volumes" Nov 01 01:10:53.623389 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:53.623358 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:54.060254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:54.060205 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:54.348137 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:54.348039 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:10:56.991828 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:10:56.991779 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:01.996177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:01.996146 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:06.362854 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:06.362821 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm"] Nov 01 01:11:06.468846 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:06.468810 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv"] Nov 01 01:11:06.917557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:06.917504 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:07.351360 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:07.351325 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:11.364018 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.363968 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" podUID="ad50d0fc-648e-477a-9b20-1585dbce5592" containerName="payload-processing" containerID="cri-o://7f77bce0d3ebc95667ced073d2040e7fae81cabc21eab7bd7a07607e44d95ac1" gracePeriod=25 Nov 01 01:11:11.469691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.469637 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" podUID="fd2997b6-e3bb-4a7a-81ce-dc62e780fc84" containerName="payload-pre-processing" containerID="cri-o://d13f5d878836884bfa353f2ee31fbc9223d682776b83cd6afc6a7cb3be6be76d" gracePeriod=25 Nov 01 01:11:11.597368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.597327 6036 generic.go:356] "Generic (PLEG): container finished" podID="fd2997b6-e3bb-4a7a-81ce-dc62e780fc84" containerID="d13f5d878836884bfa353f2ee31fbc9223d682776b83cd6afc6a7cb3be6be76d" exitCode=0 Nov 01 01:11:11.597515 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.597388 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" event={"ID":"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84","Type":"ContainerDied","Data":"d13f5d878836884bfa353f2ee31fbc9223d682776b83cd6afc6a7cb3be6be76d"} Nov 01 01:11:11.598990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.598964 6036 generic.go:356] "Generic (PLEG): container finished" podID="ad50d0fc-648e-477a-9b20-1585dbce5592" containerID="7f77bce0d3ebc95667ced073d2040e7fae81cabc21eab7bd7a07607e44d95ac1" exitCode=0 Nov 01 01:11:11.599120 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.599013 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" event={"ID":"ad50d0fc-648e-477a-9b20-1585dbce5592","Type":"ContainerDied","Data":"7f77bce0d3ebc95667ced073d2040e7fae81cabc21eab7bd7a07607e44d95ac1"} Nov 01 01:11:11.622364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.622279 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:11:11.737729 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.737702 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:11:11.743185 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.743109 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad50d0fc-648e-477a-9b20-1585dbce5592-kube-api-access-hnw2q\" (UniqueName: \"kubernetes.io/projected/ad50d0fc-648e-477a-9b20-1585dbce5592-kube-api-access-hnw2q\") pod \"ad50d0fc-648e-477a-9b20-1585dbce5592\" (UID: \"ad50d0fc-648e-477a-9b20-1585dbce5592\") " Nov 01 01:11:11.748252 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.743955 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad50d0fc-648e-477a-9b20-1585dbce5592-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50d0fc-648e-477a-9b20-1585dbce5592-plugins-config-volume\") pod \"ad50d0fc-648e-477a-9b20-1585dbce5592\" (UID: \"ad50d0fc-648e-477a-9b20-1585dbce5592\") " Nov 01 01:11:11.748252 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.744833 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad50d0fc-648e-477a-9b20-1585dbce5592-plugins-config-volume" pod "ad50d0fc-648e-477a-9b20-1585dbce5592" (UID: "ad50d0fc-648e-477a-9b20-1585dbce5592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:11:11.748252 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.745000 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50d0fc-648e-477a-9b20-1585dbce5592-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:11.748252 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.747160 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad50d0fc-648e-477a-9b20-1585dbce5592-kube-api-access-hnw2q" pod "ad50d0fc-648e-477a-9b20-1585dbce5592" (UID: "ad50d0fc-648e-477a-9b20-1585dbce5592"). InnerVolumeSpecName "kube-api-access-hnw2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:11.846246 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.846215 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-plugins-config-volume\") pod \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\" (UID: \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\") " Nov 01 01:11:11.846433 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.846385 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-kube-api-access-mgrvp\" (UniqueName: \"kubernetes.io/projected/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-kube-api-access-mgrvp\") pod \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\" (UID: \"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84\") " Nov 01 01:11:11.846623 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.846600 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-plugins-config-volume" pod "fd2997b6-e3bb-4a7a-81ce-dc62e780fc84" (UID: "fd2997b6-e3bb-4a7a-81ce-dc62e780fc84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:11:11.846765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.846752 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnw2q\" (UniqueName: \"kubernetes.io/projected/ad50d0fc-648e-477a-9b20-1585dbce5592-kube-api-access-hnw2q\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:11.846804 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.846769 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:11.848550 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.848529 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-kube-api-access-mgrvp" pod "fd2997b6-e3bb-4a7a-81ce-dc62e780fc84" (UID: "fd2997b6-e3bb-4a7a-81ce-dc62e780fc84"). InnerVolumeSpecName "kube-api-access-mgrvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:11.948312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:11.948213 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgrvp\" (UniqueName: \"kubernetes.io/projected/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84-kube-api-access-mgrvp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:12.562920 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.562888 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:12.606183 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.606151 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" Nov 01 01:11:12.606417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.606154 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv" event={"ID":"fd2997b6-e3bb-4a7a-81ce-dc62e780fc84","Type":"ContainerDied","Data":"817151b5885907ff277ecae9617e3807893fc189077498a452f78f6ba1ed0a76"} Nov 01 01:11:12.606417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.606316 6036 scope.go:122] "RemoveContainer" containerID="d13f5d878836884bfa353f2ee31fbc9223d682776b83cd6afc6a7cb3be6be76d" Nov 01 01:11:12.607733 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.607711 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" Nov 01 01:11:12.607840 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.607749 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm" event={"ID":"ad50d0fc-648e-477a-9b20-1585dbce5592","Type":"ContainerDied","Data":"cf2a077a07514dd9ec9dd4927b624a36b3fd3124e714023d0ce3c9c7c494b6c1"} Nov 01 01:11:12.617437 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.617417 6036 scope.go:122] "RemoveContainer" containerID="7f77bce0d3ebc95667ced073d2040e7fae81cabc21eab7bd7a07607e44d95ac1" Nov 01 01:11:12.774625 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.774537 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm"] Nov 01 01:11:12.781161 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.781133 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbea8a-698f75c488-pwzmm"] Nov 01 01:11:12.895058 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.895026 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv"] Nov 01 01:11:12.900421 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:12.900393 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbea8a-5cdbd6664c-9rwrv"] Nov 01 01:11:13.238102 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:13.238066 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:13.249928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:13.249892 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad50d0fc-648e-477a-9b20-1585dbce5592" path="/var/lib/kubelet/pods/ad50d0fc-648e-477a-9b20-1585dbce5592/volumes" Nov 01 01:11:13.250433 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:13.250412 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2997b6-e3bb-4a7a-81ce-dc62e780fc84" path="/var/lib/kubelet/pods/fd2997b6-e3bb-4a7a-81ce-dc62e780fc84/volumes" Nov 01 01:11:13.564673 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:13.564641 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:16.019842 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:16.019807 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:16.948336 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:16.948251 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:17.468161 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:17.468128 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:19.255113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:19.255044 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:19.508949 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:11:19.508859 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:11:19.882797 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:19.882767 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:20.398586 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:20.398556 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:23.952040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:23.952007 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:24.338613 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:24.338585 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:25.037204 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:25.037171 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:27.127804 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:27.127767 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:27.398322 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:27.398213 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:27.662029 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:27.661924 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:30.401628 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:30.401596 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:31.850695 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:31.850654 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82"] Nov 01 01:11:31.851257 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:31.850912 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" containerID="cri-o://f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38" gracePeriod=30 Nov 01 01:11:35.931672 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:35.931639 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:36.806153 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:36.806122 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:37.118078 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.118048 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 01:11:37.155478 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155445 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a0e24d-d19e-4750-9423-b6f61f034983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a0e24d-d19e-4750-9423-b6f61f034983-istiod-ca-cert\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.155639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155495 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-kube-api-access-fw2bc\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-kube-api-access-fw2bc\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.155639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155620 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-envoy\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.155737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155648 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-socket\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.155737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155691 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-certs\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.155737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155732 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-data\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.155888 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155761 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-credential-socket\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.155888 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155795 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-istio-token\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-istio-token\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.155888 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155838 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a0e24d-d19e-4750-9423-b6f61f034983-istiod-ca-cert" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:11:37.155888 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155851 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26a0e24d-d19e-4750-9423-b6f61f034983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a0e24d-d19e-4750-9423-b6f61f034983-istio-podinfo\") pod \"26a0e24d-d19e-4750-9423-b6f61f034983\" (UID: \"26a0e24d-d19e-4750-9423-b6f61f034983\") " Nov 01 01:11:37.156107 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.155951 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-socket" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:37.156107 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.156015 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-data" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:37.156107 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.156035 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-certs" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:37.156264 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.156124 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-credential-socket" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:37.156634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.156560 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.156634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.156585 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.156634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.156595 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.156634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.156605 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.156634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.156614 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a0e24d-d19e-4750-9423-b6f61f034983-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.158012 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.157987 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-envoy" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:37.158126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.158108 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26a0e24d-d19e-4750-9423-b6f61f034983-istio-podinfo" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:11:37.158526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.158506 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-istio-token" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:37.158600 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.158536 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-kube-api-access-fw2bc" pod "26a0e24d-d19e-4750-9423-b6f61f034983" (UID: "26a0e24d-d19e-4750-9423-b6f61f034983"). InnerVolumeSpecName "kube-api-access-fw2bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:37.257907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.257620 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2bc\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-kube-api-access-fw2bc\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.257907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.257643 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a0e24d-d19e-4750-9423-b6f61f034983-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.257907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.257653 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a0e24d-d19e-4750-9423-b6f61f034983-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.257907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.257662 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a0e24d-d19e-4750-9423-b6f61f034983-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:37.766664 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.766631 6036 generic.go:356] "Generic (PLEG): container finished" podID="26a0e24d-d19e-4750-9423-b6f61f034983" containerID="f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38" exitCode=0 Nov 01 01:11:37.766862 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.766742 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" Nov 01 01:11:37.766862 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.766769 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" event={"ID":"26a0e24d-d19e-4750-9423-b6f61f034983","Type":"ContainerDied","Data":"f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38"} Nov 01 01:11:37.766862 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.766824 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82" event={"ID":"26a0e24d-d19e-4750-9423-b6f61f034983","Type":"ContainerDied","Data":"03791406567d65aa95ba906cc441c63cdc2b622eac3ec7fd2b2bca5ac894c91a"} Nov 01 01:11:37.766862 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.766846 6036 scope.go:122] "RemoveContainer" containerID="f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38" Nov 01 01:11:37.780119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.780103 6036 scope.go:122] "RemoveContainer" containerID="17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c" Nov 01 01:11:37.789316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.789269 6036 scope.go:122] "RemoveContainer" containerID="f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38" Nov 01 01:11:37.789624 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:11:37.789600 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38\": container with ID starting with f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38 not found: ID does not exist" containerID="f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38" Nov 01 01:11:37.789696 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.789635 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38"} err="failed to get container status \"f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38\": rpc error: code = NotFound desc = could not find container \"f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38\": container with ID starting with f5c9d1301e73f11156fb5fa4d930a474c79e0ba5421d5aae0496704e0492fa38 not found: ID does not exist" Nov 01 01:11:37.789696 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.789654 6036 scope.go:122] "RemoveContainer" containerID="17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c" Nov 01 01:11:37.789915 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:11:37.789896 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c\": container with ID starting with 17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c not found: ID does not exist" containerID="17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c" Nov 01 01:11:37.789984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.789926 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c"} err="failed to get container status \"17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c\": rpc error: code = NotFound desc = could not find container \"17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c\": container with ID starting with 17e2191f3c7a40a3991883de65216ca14e8c2e2ea105f10c4a357b9cab30e53c not found: ID does not exist" Nov 01 01:11:37.935607 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.935571 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82"] Nov 01 01:11:37.986368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:37.986333 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-6vh82"] Nov 01 01:11:39.251068 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:39.251032 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" path="/var/lib/kubelet/pods/26a0e24d-d19e-4750-9423-b6f61f034983/volumes" Nov 01 01:11:40.345800 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.345768 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg"] Nov 01 01:11:40.346225 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.346013 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" podUID="0685b340-bbbd-4d7b-874f-c3a1cacb0310" containerName="maas-api" containerID="cri-o://2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc" gracePeriod=30 Nov 01 01:11:40.643878 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.643516 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:11:40.644434 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.644409 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:40.699988 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.699957 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls\") pod \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " Nov 01 01:11:40.700169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.700033 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0685b340-bbbd-4d7b-874f-c3a1cacb0310-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0685b340-bbbd-4d7b-874f-c3a1cacb0310-openshift-service-ca\") pod \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " Nov 01 01:11:40.700169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.700074 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-metrics-certs\") pod \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " Nov 01 01:11:40.700169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.700126 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0685b340-bbbd-4d7b-874f-c3a1cacb0310-kube-api-access-bmqmh\" (UniqueName: \"kubernetes.io/projected/0685b340-bbbd-4d7b-874f-c3a1cacb0310-kube-api-access-bmqmh\") pod \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\" (UID: \"0685b340-bbbd-4d7b-874f-c3a1cacb0310\") " Nov 01 01:11:40.700425 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.700399 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0685b340-bbbd-4d7b-874f-c3a1cacb0310-openshift-service-ca" pod "0685b340-bbbd-4d7b-874f-c3a1cacb0310" (UID: "0685b340-bbbd-4d7b-874f-c3a1cacb0310"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:11:40.700567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.700556 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0685b340-bbbd-4d7b-874f-c3a1cacb0310-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:40.702414 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.702390 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0685b340-bbbd-4d7b-874f-c3a1cacb0310-kube-api-access-bmqmh" pod "0685b340-bbbd-4d7b-874f-c3a1cacb0310" (UID: "0685b340-bbbd-4d7b-874f-c3a1cacb0310"). InnerVolumeSpecName "kube-api-access-bmqmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:40.702414 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.702404 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-metrics-certs" pod "0685b340-bbbd-4d7b-874f-c3a1cacb0310" (UID: "0685b340-bbbd-4d7b-874f-c3a1cacb0310"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:11:40.702589 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.702576 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls" pod "0685b340-bbbd-4d7b-874f-c3a1cacb0310" (UID: "0685b340-bbbd-4d7b-874f-c3a1cacb0310"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:11:40.785774 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.785740 6036 generic.go:356] "Generic (PLEG): container finished" podID="0685b340-bbbd-4d7b-874f-c3a1cacb0310" containerID="2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc" exitCode=0 Nov 01 01:11:40.785967 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.785856 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" Nov 01 01:11:40.785967 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.785872 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" event={"ID":"0685b340-bbbd-4d7b-874f-c3a1cacb0310","Type":"ContainerDied","Data":"2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc"} Nov 01 01:11:40.785967 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.785962 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg" event={"ID":"0685b340-bbbd-4d7b-874f-c3a1cacb0310","Type":"ContainerDied","Data":"d54e1f98681d9ce3f4db7ac350d03514775dec5d2a8877684024756bfee1737e"} Nov 01 01:11:40.786121 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.785984 6036 scope.go:122] "RemoveContainer" containerID="2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc" Nov 01 01:11:40.798708 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.798688 6036 scope.go:122] "RemoveContainer" containerID="2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc" Nov 01 01:11:40.799019 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:11:40.798996 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc\": container with ID starting with 2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc not found: ID does not exist" containerID="2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc" Nov 01 01:11:40.799115 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.799026 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc"} err="failed to get container status \"2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc\": rpc error: code = NotFound desc = could not find container \"2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc\": container with ID starting with 2688f12af79afd330f815118a813d663f6c8626a459bab6b00b333eaa2c145cc not found: ID does not exist" Nov 01 01:11:40.802043 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.802015 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:40.802138 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.802049 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0685b340-bbbd-4d7b-874f-c3a1cacb0310-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:40.802138 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.802067 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqmh\" (UniqueName: \"kubernetes.io/projected/0685b340-bbbd-4d7b-874f-c3a1cacb0310-kube-api-access-bmqmh\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:40.952760 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.952673 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg"] Nov 01 01:11:40.955369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:40.955342 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-030da8-5596d6d666-dckmg"] Nov 01 01:11:41.250029 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:41.249943 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0685b340-bbbd-4d7b-874f-c3a1cacb0310" path="/var/lib/kubelet/pods/0685b340-bbbd-4d7b-874f-c3a1cacb0310/volumes" Nov 01 01:11:41.474310 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:41.464832 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:41.679925 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:41.679888 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:42.147384 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:42.147330 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:42.485904 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:42.485817 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:42.737961 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:42.737875 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:43.130252 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:43.130213 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:46.802645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:46.802615 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp"] Nov 01 01:11:46.803135 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:46.802834 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" podUID="7a961708-53e1-444c-ac49-db165a3acf56" containerName="maas-api" containerID="cri-o://8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27" gracePeriod=30 Nov 01 01:11:46.965890 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:46.965859 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:47.076207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.076180 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 01:11:47.182867 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.182788 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a961708-53e1-444c-ac49-db165a3acf56-kube-api-access-67n9d\" (UniqueName: \"kubernetes.io/projected/7a961708-53e1-444c-ac49-db165a3acf56-kube-api-access-67n9d\") pod \"7a961708-53e1-444c-ac49-db165a3acf56\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " Nov 01 01:11:47.183087 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.182880 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-metrics-certs\") pod \"7a961708-53e1-444c-ac49-db165a3acf56\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " Nov 01 01:11:47.183087 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.182996 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a961708-53e1-444c-ac49-db165a3acf56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a961708-53e1-444c-ac49-db165a3acf56-openshift-service-ca\") pod \"7a961708-53e1-444c-ac49-db165a3acf56\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " Nov 01 01:11:47.183087 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.183065 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls\") pod \"7a961708-53e1-444c-ac49-db165a3acf56\" (UID: \"7a961708-53e1-444c-ac49-db165a3acf56\") " Nov 01 01:11:47.188452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.188393 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a961708-53e1-444c-ac49-db165a3acf56-openshift-service-ca" pod "7a961708-53e1-444c-ac49-db165a3acf56" (UID: "7a961708-53e1-444c-ac49-db165a3acf56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:11:47.207851 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.207803 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls" pod "7a961708-53e1-444c-ac49-db165a3acf56" (UID: "7a961708-53e1-444c-ac49-db165a3acf56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:11:47.207851 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.207828 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-metrics-certs" pod "7a961708-53e1-444c-ac49-db165a3acf56" (UID: "7a961708-53e1-444c-ac49-db165a3acf56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:11:47.208445 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.208420 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a961708-53e1-444c-ac49-db165a3acf56-kube-api-access-67n9d" pod "7a961708-53e1-444c-ac49-db165a3acf56" (UID: "7a961708-53e1-444c-ac49-db165a3acf56"). InnerVolumeSpecName "kube-api-access-67n9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:47.289526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.289484 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a961708-53e1-444c-ac49-db165a3acf56-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:47.289526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.289518 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:47.289526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.289531 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67n9d\" (UniqueName: \"kubernetes.io/projected/7a961708-53e1-444c-ac49-db165a3acf56-kube-api-access-67n9d\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:47.289771 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.289546 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a961708-53e1-444c-ac49-db165a3acf56-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:47.840713 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.840677 6036 generic.go:356] "Generic (PLEG): container finished" podID="7a961708-53e1-444c-ac49-db165a3acf56" containerID="8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27" exitCode=0 Nov 01 01:11:47.841123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.840728 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" Nov 01 01:11:47.841123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.840763 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" event={"ID":"7a961708-53e1-444c-ac49-db165a3acf56","Type":"ContainerDied","Data":"8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27"} Nov 01 01:11:47.841123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.840824 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp" event={"ID":"7a961708-53e1-444c-ac49-db165a3acf56","Type":"ContainerDied","Data":"dd0b35161ff1d183257abb6c01899ca266ddd63af8f3da5fb2199e2b0d33abb0"} Nov 01 01:11:47.841123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.840850 6036 scope.go:122] "RemoveContainer" containerID="8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27" Nov 01 01:11:47.850165 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.850146 6036 scope.go:122] "RemoveContainer" containerID="8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27" Nov 01 01:11:47.850481 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:11:47.850461 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27\": container with ID starting with 8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27 not found: ID does not exist" containerID="8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27" Nov 01 01:11:47.850562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:47.850490 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27"} err="failed to get container status \"8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27\": rpc error: code = NotFound desc = could not find container \"8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27\": container with ID starting with 8dc07416362e45cca3e00bdd03ba4950ea2243e9a2439590e357725365be9e27 not found: ID does not exist" Nov 01 01:11:48.008247 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:48.008215 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp"] Nov 01 01:11:48.010110 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:48.010085 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5a9c19-764dfb7894-5vpfp"] Nov 01 01:11:48.542856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:48.542820 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:48.765578 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:48.765547 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:49.022148 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:49.022028 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:49.250368 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:49.250331 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a961708-53e1-444c-ac49-db165a3acf56" path="/var/lib/kubelet/pods/7a961708-53e1-444c-ac49-db165a3acf56/volumes" Nov 01 01:11:49.481187 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:49.481150 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:49.504599 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:11:49.504566 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:11:49.762785 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:49.762697 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:49.988304 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:49.988262 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:50.878686 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:50.878634 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:51.589913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:51.589863 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:51.878119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:51.878034 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:53.298088 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.298048 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2"] Nov 01 01:11:53.298644 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.298329 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" podUID="fd1d1a84-c47b-4c0e-b994-e26325e429cd" containerName="maas-api" containerID="cri-o://7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f" gracePeriod=30 Nov 01 01:11:53.515797 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.515760 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:53.567907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.567885 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:11:53.669190 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.669159 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1d1a84-c47b-4c0e-b994-e26325e429cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d1a84-c47b-4c0e-b994-e26325e429cd-openshift-service-ca\") pod \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " Nov 01 01:11:53.669190 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.669192 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls\") pod \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " Nov 01 01:11:53.669452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.669231 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-metrics-certs\") pod \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " Nov 01 01:11:53.669452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.669368 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1d1a84-c47b-4c0e-b994-e26325e429cd-kube-api-access-t6zn6\" (UniqueName: \"kubernetes.io/projected/fd1d1a84-c47b-4c0e-b994-e26325e429cd-kube-api-access-t6zn6\") pod \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\" (UID: \"fd1d1a84-c47b-4c0e-b994-e26325e429cd\") " Nov 01 01:11:53.669618 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.669590 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1d1a84-c47b-4c0e-b994-e26325e429cd-openshift-service-ca" pod "fd1d1a84-c47b-4c0e-b994-e26325e429cd" (UID: "fd1d1a84-c47b-4c0e-b994-e26325e429cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:11:53.669764 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.669749 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1d1a84-c47b-4c0e-b994-e26325e429cd-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:53.671542 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.671515 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-metrics-certs" pod "fd1d1a84-c47b-4c0e-b994-e26325e429cd" (UID: "fd1d1a84-c47b-4c0e-b994-e26325e429cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:11:53.671655 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.671551 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls" pod "fd1d1a84-c47b-4c0e-b994-e26325e429cd" (UID: "fd1d1a84-c47b-4c0e-b994-e26325e429cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:11:53.671655 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.671623 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1d1a84-c47b-4c0e-b994-e26325e429cd-kube-api-access-t6zn6" pod "fd1d1a84-c47b-4c0e-b994-e26325e429cd" (UID: "fd1d1a84-c47b-4c0e-b994-e26325e429cd"). InnerVolumeSpecName "kube-api-access-t6zn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:53.770522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.770489 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6zn6\" (UniqueName: \"kubernetes.io/projected/fd1d1a84-c47b-4c0e-b994-e26325e429cd-kube-api-access-t6zn6\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:53.770522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.770521 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:53.770728 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.770534 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1d1a84-c47b-4c0e-b994-e26325e429cd-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:53.877234 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.877150 6036 generic.go:356] "Generic (PLEG): container finished" podID="fd1d1a84-c47b-4c0e-b994-e26325e429cd" containerID="7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f" exitCode=0 Nov 01 01:11:53.877234 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.877202 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" Nov 01 01:11:53.877504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.877233 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" event={"ID":"fd1d1a84-c47b-4c0e-b994-e26325e429cd","Type":"ContainerDied","Data":"7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f"} Nov 01 01:11:53.877504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.877317 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2" event={"ID":"fd1d1a84-c47b-4c0e-b994-e26325e429cd","Type":"ContainerDied","Data":"03ca1e06d048b60f86b90c6708749fa73f328665f61f302f88f440aa5e205750"} Nov 01 01:11:53.877504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.877343 6036 scope.go:122] "RemoveContainer" containerID="7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f" Nov 01 01:11:53.887674 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.887656 6036 scope.go:122] "RemoveContainer" containerID="7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f" Nov 01 01:11:53.887971 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:11:53.887949 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f\": container with ID starting with 7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f not found: ID does not exist" containerID="7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f" Nov 01 01:11:53.888020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:53.887979 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f"} err="failed to get container status \"7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f\": rpc error: code = NotFound desc = could not find container \"7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f\": container with ID starting with 7fe923fdf6578fc21daeac3dd83f71cb3337e9722e4b87376396bc11d4485e9f not found: ID does not exist" Nov 01 01:11:54.047269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:54.047238 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2"] Nov 01 01:11:54.050791 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:54.050767 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbea8a-784947f494-hnxh2"] Nov 01 01:11:55.131988 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:55.131932 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:55.252592 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:55.252559 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1d1a84-c47b-4c0e-b994-e26325e429cd" path="/var/lib/kubelet/pods/fd1d1a84-c47b-4c0e-b994-e26325e429cd/volumes" Nov 01 01:11:55.423508 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:55.423374 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:55.661861 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:55.661821 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:56.142522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:56.142462 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:56.452744 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:56.452464 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:56.751610 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:56.751472 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:57.196279 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:57.196222 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:57.460057 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:57.459937 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:11:57.695151 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:11:57.695107 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:00.472753 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:00.472699 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:00.689948 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:00.689901 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:01.085960 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:01.085925 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:03.068486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:03.068408 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:03.289451 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:03.289399 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:03.607249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:03.607211 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:03.999224 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:03.999111 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:06.249776 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:06.249740 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:08.889021 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:08.888981 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9"] Nov 01 01:12:08.939918 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:08.939883 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz"] Nov 01 01:12:08.941825 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:08.941797 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2"] Nov 01 01:12:09.122062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:09.122032 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:11.486572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:11.486534 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:15.540903 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.540866 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz"] Nov 01 01:12:15.541354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.541129 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" podUID="50a18b44-be40-48e3-b4f3-383170ca5c7a" containerName="maas-api" containerID="cri-o://58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd" gracePeriod=30 Nov 01 01:12:15.717439 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.717367 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:15.808810 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.808752 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:12:15.935779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.935740 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a18b44-be40-48e3-b4f3-383170ca5c7a-kube-api-access-kpsqx\" (UniqueName: \"kubernetes.io/projected/50a18b44-be40-48e3-b4f3-383170ca5c7a-kube-api-access-kpsqx\") pod \"50a18b44-be40-48e3-b4f3-383170ca5c7a\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " Nov 01 01:12:15.936838 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.936812 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-metrics-certs\") pod \"50a18b44-be40-48e3-b4f3-383170ca5c7a\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " Nov 01 01:12:15.937054 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.937037 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls\") pod \"50a18b44-be40-48e3-b4f3-383170ca5c7a\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " Nov 01 01:12:15.937227 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.937209 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a18b44-be40-48e3-b4f3-383170ca5c7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a18b44-be40-48e3-b4f3-383170ca5c7a-openshift-service-ca\") pod \"50a18b44-be40-48e3-b4f3-383170ca5c7a\" (UID: \"50a18b44-be40-48e3-b4f3-383170ca5c7a\") " Nov 01 01:12:15.940002 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.939975 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a18b44-be40-48e3-b4f3-383170ca5c7a-openshift-service-ca" pod "50a18b44-be40-48e3-b4f3-383170ca5c7a" (UID: "50a18b44-be40-48e3-b4f3-383170ca5c7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:15.962567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.962491 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a18b44-be40-48e3-b4f3-383170ca5c7a-kube-api-access-kpsqx" pod "50a18b44-be40-48e3-b4f3-383170ca5c7a" (UID: "50a18b44-be40-48e3-b4f3-383170ca5c7a"). InnerVolumeSpecName "kube-api-access-kpsqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:15.963700 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.963665 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-metrics-certs" pod "50a18b44-be40-48e3-b4f3-383170ca5c7a" (UID: "50a18b44-be40-48e3-b4f3-383170ca5c7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:15.964331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:15.964302 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls" pod "50a18b44-be40-48e3-b4f3-383170ca5c7a" (UID: "50a18b44-be40-48e3-b4f3-383170ca5c7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:16.006635 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.006604 6036 generic.go:356] "Generic (PLEG): container finished" podID="50a18b44-be40-48e3-b4f3-383170ca5c7a" containerID="58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd" exitCode=0 Nov 01 01:12:16.006805 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.006664 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" Nov 01 01:12:16.006805 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.006698 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" event={"ID":"50a18b44-be40-48e3-b4f3-383170ca5c7a","Type":"ContainerDied","Data":"58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd"} Nov 01 01:12:16.006805 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.006777 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz" event={"ID":"50a18b44-be40-48e3-b4f3-383170ca5c7a","Type":"ContainerDied","Data":"bf9f2e8642708d51de77cd3019ab1135615ecb9b470b9dd389416d5c3a311f27"} Nov 01 01:12:16.006805 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.006801 6036 scope.go:122] "RemoveContainer" containerID="58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd" Nov 01 01:12:16.017126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.017103 6036 scope.go:122] "RemoveContainer" containerID="58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd" Nov 01 01:12:16.017419 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:16.017388 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd\": container with ID starting with 58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd not found: ID does not exist" containerID="58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd" Nov 01 01:12:16.017474 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.017421 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd"} err="failed to get container status \"58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd\": rpc error: code = NotFound desc = could not find container \"58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd\": container with ID starting with 58598ba04b74bf291bb11c44e1bcd7bbe95fdff815150f5b4ac7ae8c6f40febd not found: ID does not exist" Nov 01 01:12:16.041137 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.041109 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpsqx\" (UniqueName: \"kubernetes.io/projected/50a18b44-be40-48e3-b4f3-383170ca5c7a-kube-api-access-kpsqx\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:16.041137 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.041135 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:16.041332 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.041147 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a18b44-be40-48e3-b4f3-383170ca5c7a-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:16.041332 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.041155 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a18b44-be40-48e3-b4f3-383170ca5c7a-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:16.176344 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.176246 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz"] Nov 01 01:12:16.180144 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.180119 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1402e8-6f798896b8-x77hz"] Nov 01 01:12:16.572809 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.572770 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:16.792347 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:16.792313 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:17.250064 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:17.249964 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a18b44-be40-48e3-b4f3-383170ca5c7a" path="/var/lib/kubelet/pods/50a18b44-be40-48e3-b4f3-383170ca5c7a/volumes" Nov 01 01:12:17.270690 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:17.270635 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:17.483682 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:17.483633 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:17.722836 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:17.722802 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:18.177698 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:18.177669 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:19.507804 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:19.507765 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:12:23.050955 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:23.050923 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:23.910579 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:23.910539 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" containerID="cri-o://26a81659fb46a8b41b87f5769c54badb5633e94f0dd6ba33d9ef79ea7ef0f2a5" gracePeriod=45 Nov 01 01:12:23.960805 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:23.960771 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" containerID="cri-o://4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051" gracePeriod=45 Nov 01 01:12:23.962030 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:23.961992 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" containerID="cri-o://5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e" gracePeriod=45 Nov 01 01:12:24.054602 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.054567 6036 generic.go:356] "Generic (PLEG): container finished" podID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerID="26a81659fb46a8b41b87f5769c54badb5633e94f0dd6ba33d9ef79ea7ef0f2a5" exitCode=0 Nov 01 01:12:24.055006 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.054643 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" event={"ID":"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d","Type":"ContainerDied","Data":"26a81659fb46a8b41b87f5769c54badb5633e94f0dd6ba33d9ef79ea7ef0f2a5"} Nov 01 01:12:24.111970 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.111925 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": dial tcp 10.134.0.62:8000: connect: connection refused" Nov 01 01:12:24.242065 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.238770 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 01:12:24.264462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.264436 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 01:12:24.268844 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.268823 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 01:12:24.348538 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348503 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tls-certs\") pod \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " Nov 01 01:12:24.348718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348556 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-model-cache\") pod \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " Nov 01 01:12:24.348718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348587 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-dshm\") pod \"f75a426a-af3c-4361-b7ba-a3f06345820f\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " Nov 01 01:12:24.348718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348633 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-model-cache\") pod \"f75a426a-af3c-4361-b7ba-a3f06345820f\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " Nov 01 01:12:24.348718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348652 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-model-cache\") pod \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " Nov 01 01:12:24.348718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348693 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-dshm\") pod \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " Nov 01 01:12:24.349025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348742 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-home\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-home\") pod \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " Nov 01 01:12:24.349025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348768 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-dshm\") pod \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " Nov 01 01:12:24.349025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348811 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f75a426a-af3c-4361-b7ba-a3f06345820f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f75a426a-af3c-4361-b7ba-a3f06345820f-tls-certs\") pod \"f75a426a-af3c-4361-b7ba-a3f06345820f\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " Nov 01 01:12:24.349025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348844 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-model-cache" pod "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" (UID: "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.349025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348854 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-tmp-dir\") pod \"f75a426a-af3c-4361-b7ba-a3f06345820f\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " Nov 01 01:12:24.349025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.348975 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f75a426a-af3c-4361-b7ba-a3f06345820f-kube-api-access-8zbq8\" (UniqueName: \"kubernetes.io/projected/f75a426a-af3c-4361-b7ba-a3f06345820f-kube-api-access-8zbq8\") pod \"f75a426a-af3c-4361-b7ba-a3f06345820f\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " Nov 01 01:12:24.349025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349007 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-kube-api-access-nbkrz\" (UniqueName: \"kubernetes.io/projected/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-kube-api-access-nbkrz\") pod \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " Nov 01 01:12:24.349025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349028 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-kube-api-access-bzmjx\" (UniqueName: \"kubernetes.io/projected/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-kube-api-access-bzmjx\") pod \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " Nov 01 01:12:24.349448 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349068 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tmp-dir\") pod \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " Nov 01 01:12:24.349448 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349184 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-home" pod "4166a66b-53e0-4afa-adf6-02b33d0f8ec1" (UID: "4166a66b-53e0-4afa-adf6-02b33d0f8ec1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.349576 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349527 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-home\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-home\") pod \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " Nov 01 01:12:24.349576 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349561 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tmp-dir\") pod \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\" (UID: \"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d\") " Nov 01 01:12:24.349698 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349587 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-model-cache" pod "f75a426a-af3c-4361-b7ba-a3f06345820f" (UID: "f75a426a-af3c-4361-b7ba-a3f06345820f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.349698 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349658 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tls-certs\") pod \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\" (UID: \"4166a66b-53e0-4afa-adf6-02b33d0f8ec1\") " Nov 01 01:12:24.349915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.349851 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-home\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-home\") pod \"f75a426a-af3c-4361-b7ba-a3f06345820f\" (UID: \"f75a426a-af3c-4361-b7ba-a3f06345820f\") " Nov 01 01:12:24.350240 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.350205 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-tmp-dir" pod "f75a426a-af3c-4361-b7ba-a3f06345820f" (UID: "f75a426a-af3c-4361-b7ba-a3f06345820f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.350793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.350469 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.350793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.350492 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.350793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.350507 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.350793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.350520 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.350793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.350560 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tmp-dir" pod "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" (UID: "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.350793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.350616 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-home" pod "f75a426a-af3c-4361-b7ba-a3f06345820f" (UID: "f75a426a-af3c-4361-b7ba-a3f06345820f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.350793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.350725 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-home" pod "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" (UID: "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.351503 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.351299 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tmp-dir" pod "4166a66b-53e0-4afa-adf6-02b33d0f8ec1" (UID: "4166a66b-53e0-4afa-adf6-02b33d0f8ec1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.351503 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.351478 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-model-cache" pod "4166a66b-53e0-4afa-adf6-02b33d0f8ec1" (UID: "4166a66b-53e0-4afa-adf6-02b33d0f8ec1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.351781 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.351753 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tls-certs" pod "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" (UID: "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:24.352557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.352534 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-dshm" pod "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" (UID: "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.354073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.354044 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75a426a-af3c-4361-b7ba-a3f06345820f-kube-api-access-8zbq8" pod "f75a426a-af3c-4361-b7ba-a3f06345820f" (UID: "f75a426a-af3c-4361-b7ba-a3f06345820f"). InnerVolumeSpecName "kube-api-access-8zbq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:24.354073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.354044 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-dshm" pod "f75a426a-af3c-4361-b7ba-a3f06345820f" (UID: "f75a426a-af3c-4361-b7ba-a3f06345820f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.354211 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.354164 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-kube-api-access-nbkrz" pod "4166a66b-53e0-4afa-adf6-02b33d0f8ec1" (UID: "4166a66b-53e0-4afa-adf6-02b33d0f8ec1"). InnerVolumeSpecName "kube-api-access-nbkrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:24.354409 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.354223 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f75a426a-af3c-4361-b7ba-a3f06345820f-tls-certs" pod "f75a426a-af3c-4361-b7ba-a3f06345820f" (UID: "f75a426a-af3c-4361-b7ba-a3f06345820f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:24.354792 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.354765 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-kube-api-access-bzmjx" pod "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" (UID: "bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d"). InnerVolumeSpecName "kube-api-access-bzmjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:24.355719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.355690 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tls-certs" pod "4166a66b-53e0-4afa-adf6-02b33d0f8ec1" (UID: "4166a66b-53e0-4afa-adf6-02b33d0f8ec1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:24.355719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.355707 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-dshm" pod "4166a66b-53e0-4afa-adf6-02b33d0f8ec1" (UID: "4166a66b-53e0-4afa-adf6-02b33d0f8ec1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:24.388626 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.388591 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:24.451254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451177 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451212 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75a426a-af3c-4361-b7ba-a3f06345820f-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451228 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbq8\" (UniqueName: \"kubernetes.io/projected/f75a426a-af3c-4361-b7ba-a3f06345820f-kube-api-access-8zbq8\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451237 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbkrz\" (UniqueName: \"kubernetes.io/projected/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-kube-api-access-nbkrz\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451247 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmjx\" (UniqueName: \"kubernetes.io/projected/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-kube-api-access-bzmjx\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451256 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451265 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451273 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451301 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451311 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451320 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451327 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75a426a-af3c-4361-b7ba-a3f06345820f-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451334 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4166a66b-53e0-4afa-adf6-02b33d0f8ec1-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.451588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.451342 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:24.679053 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.679019 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:24.991691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:24.991640 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:25.061788 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.061750 6036 generic.go:356] "Generic (PLEG): container finished" podID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerID="5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e" exitCode=0 Nov 01 01:12:25.062350 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.061822 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" Nov 01 01:12:25.062350 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.061847 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" event={"ID":"f75a426a-af3c-4361-b7ba-a3f06345820f","Type":"ContainerDied","Data":"5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e"} Nov 01 01:12:25.062350 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.061893 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2" event={"ID":"f75a426a-af3c-4361-b7ba-a3f06345820f","Type":"ContainerDied","Data":"dfbc49d6496d1bfe863b43d4daf64397560f2a114914350aefd60ccd9e422cae"} Nov 01 01:12:25.062350 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.061919 6036 scope.go:122] "RemoveContainer" containerID="5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e" Nov 01 01:12:25.063871 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.063852 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" Nov 01 01:12:25.064039 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.063868 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9" event={"ID":"bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d","Type":"ContainerDied","Data":"1dd527009ddff51def3ec6e4ef9b69c87d12cc3d70f71ac76176a7235b6fb410"} Nov 01 01:12:25.065788 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.065757 6036 generic.go:356] "Generic (PLEG): container finished" podID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerID="4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051" exitCode=0 Nov 01 01:12:25.065936 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.065816 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" event={"ID":"4166a66b-53e0-4afa-adf6-02b33d0f8ec1","Type":"ContainerDied","Data":"4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051"} Nov 01 01:12:25.065936 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.065827 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" Nov 01 01:12:25.065936 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.065842 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz" event={"ID":"4166a66b-53e0-4afa-adf6-02b33d0f8ec1","Type":"ContainerDied","Data":"3134e70bdd9b2df3a211a012e727b77ac6f3e19fee639c549299243f3f2ab1ce"} Nov 01 01:12:25.077717 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.075220 6036 scope.go:122] "RemoveContainer" containerID="5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e" Nov 01 01:12:25.077717 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:25.077433 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e\": container with ID starting with 5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e not found: ID does not exist" containerID="5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e" Nov 01 01:12:25.077717 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.077459 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e"} err="failed to get container status \"5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e\": rpc error: code = NotFound desc = could not find container \"5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e\": container with ID starting with 5eabe3c431fe82d2e90be1b99be2b1839bf1d3a7ac2b73c85644a10e5ccda81e not found: ID does not exist" Nov 01 01:12:25.077717 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.077484 6036 scope.go:122] "RemoveContainer" containerID="26a81659fb46a8b41b87f5769c54badb5633e94f0dd6ba33d9ef79ea7ef0f2a5" Nov 01 01:12:25.086558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.086540 6036 scope.go:122] "RemoveContainer" containerID="4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051" Nov 01 01:12:25.094668 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.094644 6036 scope.go:122] "RemoveContainer" containerID="4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051" Nov 01 01:12:25.094992 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:25.094963 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051\": container with ID starting with 4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051 not found: ID does not exist" containerID="4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051" Nov 01 01:12:25.095051 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.095000 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051"} err="failed to get container status \"4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051\": rpc error: code = NotFound desc = could not find container \"4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051\": container with ID starting with 4b003f63310891b85f11e45500e6c081cb5ca7d73f2652277267d72de4ae7051 not found: ID does not exist" Nov 01 01:12:25.230626 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.230540 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz"] Nov 01 01:12:25.246996 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.246958 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:25.249791 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.249767 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-w1-30fb19-kserve-7bbd6c6d5-p95dz"] Nov 01 01:12:25.376147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.376111 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2"] Nov 01 01:12:25.378744 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.378718 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/simulator-w1-30fb19-kserve-b4c576679-d7sc2"] Nov 01 01:12:25.489398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.489310 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9"] Nov 01 01:12:25.492003 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:25.491971 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-30fb19/distinct-2-w1-30fb19-kserve-56b9d46554-xz2n9"] Nov 01 01:12:27.250205 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:27.250172 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" path="/var/lib/kubelet/pods/4166a66b-53e0-4afa-adf6-02b33d0f8ec1/volumes" Nov 01 01:12:27.250637 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:27.250626 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" path="/var/lib/kubelet/pods/bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d/volumes" Nov 01 01:12:27.251057 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:27.251042 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" path="/var/lib/kubelet/pods/f75a426a-af3c-4361-b7ba-a3f06345820f/volumes" Nov 01 01:12:28.293687 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:28.293649 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:28.629357 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:28.629265 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:30.148965 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:30.148925 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq"] Nov 01 01:12:30.363673 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:30.363635 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:33.372829 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:33.372795 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:36.849908 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.849877 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z"] Nov 01 01:12:36.850417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850402 6036 cpu_manager.go:414] "containerMap: removing container" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850419 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850432 6036 cpu_manager.go:414] "containerMap: removing container" podUID="fd1d1a84-c47b-4c0e-b994-e26325e429cd" containerName="maas-api" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850443 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1d1a84-c47b-4c0e-b994-e26325e429cd" containerName="maas-api" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850453 6036 cpu_manager.go:414] "containerMap: removing container" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850458 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850474 6036 cpu_manager.go:414] "containerMap: removing container" podUID="7a961708-53e1-444c-ac49-db165a3acf56" containerName="maas-api" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850479 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a961708-53e1-444c-ac49-db165a3acf56" containerName="maas-api" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850487 6036 cpu_manager.go:414] "containerMap: removing container" podUID="ad50d0fc-648e-477a-9b20-1585dbce5592" containerName="payload-processing" Nov 01 01:12:36.850490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850491 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad50d0fc-648e-477a-9b20-1585dbce5592" containerName="payload-processing" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850499 6036 cpu_manager.go:414] "containerMap: removing container" podUID="fd2997b6-e3bb-4a7a-81ce-dc62e780fc84" containerName="payload-pre-processing" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850503 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2997b6-e3bb-4a7a-81ce-dc62e780fc84" containerName="payload-pre-processing" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850513 6036 cpu_manager.go:414] "containerMap: removing container" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850517 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850532 6036 cpu_manager.go:414] "containerMap: removing container" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850536 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850545 6036 cpu_manager.go:414] "containerMap: removing container" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850549 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850561 6036 cpu_manager.go:414] "containerMap: removing container" podUID="0685b340-bbbd-4d7b-874f-c3a1cacb0310" containerName="maas-api" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850565 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0685b340-bbbd-4d7b-874f-c3a1cacb0310" containerName="maas-api" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850571 6036 cpu_manager.go:414] "containerMap: removing container" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850575 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850599 6036 cpu_manager.go:414] "containerMap: removing container" podUID="50a18b44-be40-48e3-b4f3-383170ca5c7a" containerName="maas-api" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850603 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a18b44-be40-48e3-b4f3-383170ca5c7a" containerName="maas-api" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850611 6036 cpu_manager.go:414] "containerMap: removing container" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850615 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850699 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf73da4f-eab7-4f6b-9ecd-18e5d2d1367d" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850708 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4166a66b-53e0-4afa-adf6-02b33d0f8ec1" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850713 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1d1a84-c47b-4c0e-b994-e26325e429cd" containerName="maas-api" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850718 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850724 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a961708-53e1-444c-ac49-db165a3acf56" containerName="maas-api" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850729 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad50d0fc-648e-477a-9b20-1585dbce5592" containerName="payload-processing" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850734 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2997b6-e3bb-4a7a-81ce-dc62e780fc84" containerName="payload-pre-processing" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850739 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850745 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="244bd1e4-d0c1-45b9-a8bc-ca172ba181a6" containerName="istio-proxy" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850751 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f75a426a-af3c-4361-b7ba-a3f06345820f" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850764 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0685b340-bbbd-4d7b-874f-c3a1cacb0310" containerName="maas-api" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850769 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="828bbae3-be6b-45df-8516-9f659098e61d" containerName="main" Nov 01 01:12:36.850765 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.850777 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a18b44-be40-48e3-b4f3-383170ca5c7a" containerName="maas-api" Nov 01 01:12:36.855742 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.855723 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.909277 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.909236 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7fdf2c-openshift-default-dockercfg-dk7pr\"" Nov 01 01:12:36.962076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.962044 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z"] Nov 01 01:12:36.995155 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995119 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.995155 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995159 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.995504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995269 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.995504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995337 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.995504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995473 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.995643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995518 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.995643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995597 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.995729 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995687 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qp9\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-kube-api-access-n8qp9\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:36.995729 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:36.995708 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.096787 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.096752 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.096989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.096789 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.096989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.096840 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.096989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.096911 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qp9\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-kube-api-access-n8qp9\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.096989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.096941 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.096989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.096989 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.097249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.097008 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.097249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.097067 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.097249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.097106 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.097249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.097226 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.097571 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.097233 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.097571 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.097393 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.097697 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.097627 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.097869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.097849 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.099725 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.099693 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.100007 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.099958 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.110408 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.110381 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:37.158994 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.158953 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.159146 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.159030 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qp9\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-kube-api-access-n8qp9\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.169352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.169328 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:37.260966 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.260931 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6"] Nov 01 01:12:37.371564 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.371485 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz"] Nov 01 01:12:37.598360 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.598336 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z"] Nov 01 01:12:37.600422 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:12:37.600394 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf52ec0_ebbf_44ac_bc3e_dae9caac2d47.slice/crio-302ff0529c6fe4bfbf442334efa00d477afd79a5fcb3e883062a18ed4c5f5b66 WatchSource:0}: Error finding container 302ff0529c6fe4bfbf442334efa00d477afd79a5fcb3e883062a18ed4c5f5b66: Status 404 returned error can't find the container with id 302ff0529c6fe4bfbf442334efa00d477afd79a5fcb3e883062a18ed4c5f5b66 Nov 01 01:12:37.602711 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.602673 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:37.602810 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.602746 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:37.602810 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.602777 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:37.718235 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.718190 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:37.930270 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:37.930171 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:38.140864 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:38.140831 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" event={"ID":"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47","Type":"ContainerStarted","Data":"8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca"} Nov 01 01:12:38.141054 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:38.140890 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" event={"ID":"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47","Type":"ContainerStarted","Data":"302ff0529c6fe4bfbf442334efa00d477afd79a5fcb3e883062a18ed4c5f5b66"} Nov 01 01:12:38.170230 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:38.170200 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:38.171962 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:38.171941 6036 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 01:12:38.172033 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:38.171986 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 01:12:38.263309 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:38.263211 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:38.263930 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:38.263883 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podStartSLOduration=2.263869263 podStartE2EDuration="2.263869263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:38.262116175 +0000 UTC m=+2903.137337191" watchObservedRunningTime="2026-11-01 01:12:38.263869263 +0000 UTC m=+2903.139090277" Nov 01 01:12:39.170778 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:39.170740 6036 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 01:12:39.171350 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:39.170797 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 01:12:40.170867 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:40.170828 6036 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 01:12:40.171444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:40.170897 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 01:12:41.170733 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:41.170694 6036 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 01:12:41.170930 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:41.170766 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 01:12:42.170145 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.170102 6036 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 01:12:42.170562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.170525 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 01:12:42.268312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.266377 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" podUID="751ff7e8-5d91-4386-8882-c7d75d4e668f" containerName="payload-processing" containerID="cri-o://74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5" gracePeriod=25 Nov 01 01:12:42.383551 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.372390 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" podUID="675d3e25-4ab9-46be-878a-d049d7e50e58" containerName="payload-pre-processing" containerID="cri-o://6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c" gracePeriod=25 Nov 01 01:12:42.618709 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.618677 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 01:12:42.732620 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.732594 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 01:12:42.793264 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.793235 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751ff7e8-5d91-4386-8882-c7d75d4e668f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751ff7e8-5d91-4386-8882-c7d75d4e668f-plugins-config-volume\") pod \"751ff7e8-5d91-4386-8882-c7d75d4e668f\" (UID: \"751ff7e8-5d91-4386-8882-c7d75d4e668f\") " Nov 01 01:12:42.793480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.793360 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751ff7e8-5d91-4386-8882-c7d75d4e668f-kube-api-access-zkp7w\" (UniqueName: \"kubernetes.io/projected/751ff7e8-5d91-4386-8882-c7d75d4e668f-kube-api-access-zkp7w\") pod \"751ff7e8-5d91-4386-8882-c7d75d4e668f\" (UID: \"751ff7e8-5d91-4386-8882-c7d75d4e668f\") " Nov 01 01:12:42.793847 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.793685 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751ff7e8-5d91-4386-8882-c7d75d4e668f-plugins-config-volume" pod "751ff7e8-5d91-4386-8882-c7d75d4e668f" (UID: "751ff7e8-5d91-4386-8882-c7d75d4e668f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:42.795783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.795758 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751ff7e8-5d91-4386-8882-c7d75d4e668f-kube-api-access-zkp7w" pod "751ff7e8-5d91-4386-8882-c7d75d4e668f" (UID: "751ff7e8-5d91-4386-8882-c7d75d4e668f"). InnerVolumeSpecName "kube-api-access-zkp7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:42.894025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.893997 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675d3e25-4ab9-46be-878a-d049d7e50e58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/675d3e25-4ab9-46be-878a-d049d7e50e58-plugins-config-volume\") pod \"675d3e25-4ab9-46be-878a-d049d7e50e58\" (UID: \"675d3e25-4ab9-46be-878a-d049d7e50e58\") " Nov 01 01:12:42.894457 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.894431 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675d3e25-4ab9-46be-878a-d049d7e50e58-kube-api-access-vk76l\" (UniqueName: \"kubernetes.io/projected/675d3e25-4ab9-46be-878a-d049d7e50e58-kube-api-access-vk76l\") pod \"675d3e25-4ab9-46be-878a-d049d7e50e58\" (UID: \"675d3e25-4ab9-46be-878a-d049d7e50e58\") " Nov 01 01:12:42.894785 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.894538 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675d3e25-4ab9-46be-878a-d049d7e50e58-plugins-config-volume" pod "675d3e25-4ab9-46be-878a-d049d7e50e58" (UID: "675d3e25-4ab9-46be-878a-d049d7e50e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:42.895031 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.895018 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkp7w\" (UniqueName: \"kubernetes.io/projected/751ff7e8-5d91-4386-8882-c7d75d4e668f-kube-api-access-zkp7w\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:42.895120 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.895108 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/675d3e25-4ab9-46be-878a-d049d7e50e58-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:42.895181 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.895140 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751ff7e8-5d91-4386-8882-c7d75d4e668f-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:42.896991 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.896965 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675d3e25-4ab9-46be-878a-d049d7e50e58-kube-api-access-vk76l" pod "675d3e25-4ab9-46be-878a-d049d7e50e58" (UID: "675d3e25-4ab9-46be-878a-d049d7e50e58"). InnerVolumeSpecName "kube-api-access-vk76l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:42.995920 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:42.995839 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk76l\" (UniqueName: \"kubernetes.io/projected/675d3e25-4ab9-46be-878a-d049d7e50e58-kube-api-access-vk76l\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:43.163857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.163820 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:43.174697 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.174661 6036 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 01:12:43.174983 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.174949 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 01:12:43.195115 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.195075 6036 generic.go:356] "Generic (PLEG): container finished" podID="751ff7e8-5d91-4386-8882-c7d75d4e668f" containerID="74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5" exitCode=0 Nov 01 01:12:43.195633 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.195138 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" Nov 01 01:12:43.195633 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.195146 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" event={"ID":"751ff7e8-5d91-4386-8882-c7d75d4e668f","Type":"ContainerDied","Data":"74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5"} Nov 01 01:12:43.195633 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.195201 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6" event={"ID":"751ff7e8-5d91-4386-8882-c7d75d4e668f","Type":"ContainerDied","Data":"076e9783534a6bf30e9a7a1ef3ffa91324a5441f37d1069075c181faa68609f7"} Nov 01 01:12:43.195633 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.195223 6036 scope.go:122] "RemoveContainer" containerID="74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5" Nov 01 01:12:43.196878 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.196812 6036 generic.go:356] "Generic (PLEG): container finished" podID="675d3e25-4ab9-46be-878a-d049d7e50e58" containerID="6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c" exitCode=0 Nov 01 01:12:43.196878 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.196861 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" Nov 01 01:12:43.197013 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.196968 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" event={"ID":"675d3e25-4ab9-46be-878a-d049d7e50e58","Type":"ContainerDied","Data":"6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c"} Nov 01 01:12:43.197066 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.197011 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz" event={"ID":"675d3e25-4ab9-46be-878a-d049d7e50e58","Type":"ContainerDied","Data":"9581f662a75b908e532768c9c175e41f3ebf06c33d01a54704ac267a2981ab8b"} Nov 01 01:12:43.208728 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.208709 6036 scope.go:122] "RemoveContainer" containerID="74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5" Nov 01 01:12:43.209163 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:43.209142 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5\": container with ID starting with 74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5 not found: ID does not exist" containerID="74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5" Nov 01 01:12:43.209265 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.209174 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5"} err="failed to get container status \"74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5\": rpc error: code = NotFound desc = could not find container \"74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5\": container with ID starting with 74f6637c0cc35a01f11efbbe64bc53501e6702952e8c08d89640ba47c30932d5 not found: ID does not exist" Nov 01 01:12:43.209265 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.209199 6036 scope.go:122] "RemoveContainer" containerID="6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c" Nov 01 01:12:43.217692 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.217670 6036 scope.go:122] "RemoveContainer" containerID="6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c" Nov 01 01:12:43.217974 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:43.217954 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c\": container with ID starting with 6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c not found: ID does not exist" containerID="6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c" Nov 01 01:12:43.218015 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.217983 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c"} err="failed to get container status \"6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c\": rpc error: code = NotFound desc = could not find container \"6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c\": container with ID starting with 6cf9da8f26314198aae88c1dc56c4c8730d024930b1283ebbd02ed4a79bd858c not found: ID does not exist" Nov 01 01:12:43.373383 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.368409 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6"] Nov 01 01:12:43.373383 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.371504 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-22d7b2-68f46b9b84-cqzn6"] Nov 01 01:12:43.494430 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.493541 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz"] Nov 01 01:12:43.495341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:43.495321 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-22d7b2-8c9dc748c-ghhkz"] Nov 01 01:12:44.262416 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:44.262376 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:44.263338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:44.263275 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:44.264114 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:44.264094 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:12:45.170001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.169924 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" containerID="cri-o://67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584" gracePeriod=45 Nov 01 01:12:45.246943 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.246909 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:45.250387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.250364 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675d3e25-4ab9-46be-878a-d049d7e50e58" path="/var/lib/kubelet/pods/675d3e25-4ab9-46be-878a-d049d7e50e58/volumes" Nov 01 01:12:45.250810 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.250796 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="751ff7e8-5d91-4386-8882-c7d75d4e668f" path="/var/lib/kubelet/pods/751ff7e8-5d91-4386-8882-c7d75d4e668f/volumes" Nov 01 01:12:45.433836 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.433774 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 01:12:45.436849 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.436828 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tmp-dir\") pod \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " Nov 01 01:12:45.436950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.436858 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-model-cache\") pod \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " Nov 01 01:12:45.436950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.436916 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tls-certs\") pod \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " Nov 01 01:12:45.436950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.436942 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-dshm\") pod \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " Nov 01 01:12:45.437055 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.437009 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-home\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-home\") pod \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " Nov 01 01:12:45.437055 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.437038 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af64c3fe-059e-4f14-9eb0-910b05dae0b5-kube-api-access-gnppj\" (UniqueName: \"kubernetes.io/projected/af64c3fe-059e-4f14-9eb0-910b05dae0b5-kube-api-access-gnppj\") pod \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\" (UID: \"af64c3fe-059e-4f14-9eb0-910b05dae0b5\") " Nov 01 01:12:45.437142 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.437099 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tmp-dir" pod "af64c3fe-059e-4f14-9eb0-910b05dae0b5" (UID: "af64c3fe-059e-4f14-9eb0-910b05dae0b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.437198 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.437174 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-model-cache" pod "af64c3fe-059e-4f14-9eb0-910b05dae0b5" (UID: "af64c3fe-059e-4f14-9eb0-910b05dae0b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.437477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.437267 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-home" pod "af64c3fe-059e-4f14-9eb0-910b05dae0b5" (UID: "af64c3fe-059e-4f14-9eb0-910b05dae0b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.437595 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.437569 6036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-home\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.437676 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.437596 6036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tmp-dir\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.437676 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.437609 6036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-model-cache\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.439358 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.439337 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-dshm" pod "af64c3fe-059e-4f14-9eb0-910b05dae0b5" (UID: "af64c3fe-059e-4f14-9eb0-910b05dae0b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.439453 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.439365 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tls-certs" pod "af64c3fe-059e-4f14-9eb0-910b05dae0b5" (UID: "af64c3fe-059e-4f14-9eb0-910b05dae0b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:45.439453 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.439373 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af64c3fe-059e-4f14-9eb0-910b05dae0b5-kube-api-access-gnppj" pod "af64c3fe-059e-4f14-9eb0-910b05dae0b5" (UID: "af64c3fe-059e-4f14-9eb0-910b05dae0b5"). InnerVolumeSpecName "kube-api-access-gnppj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:45.452372 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.452337 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:45.538437 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.538406 6036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af64c3fe-059e-4f14-9eb0-910b05dae0b5-tls-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.538437 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.538432 6036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af64c3fe-059e-4f14-9eb0-910b05dae0b5-dshm\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.538437 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:45.538443 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnppj\" (UniqueName: \"kubernetes.io/projected/af64c3fe-059e-4f14-9eb0-910b05dae0b5-kube-api-access-gnppj\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:46.216588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.216552 6036 generic.go:356] "Generic (PLEG): container finished" podID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerID="67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584" exitCode=0 Nov 01 01:12:46.217113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.216593 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" Nov 01 01:12:46.217113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.216630 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" event={"ID":"af64c3fe-059e-4f14-9eb0-910b05dae0b5","Type":"ContainerDied","Data":"67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584"} Nov 01 01:12:46.217113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.216675 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq" event={"ID":"af64c3fe-059e-4f14-9eb0-910b05dae0b5","Type":"ContainerDied","Data":"8740b22fb0abd29231027250c3eb0323acc9cae8aae6c02d9ac946ea2da0667e"} Nov 01 01:12:46.217113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.216693 6036 scope.go:122] "RemoveContainer" containerID="67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584" Nov 01 01:12:46.227215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.227193 6036 scope.go:122] "RemoveContainer" containerID="67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584" Nov 01 01:12:46.227522 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:46.227504 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584\": container with ID starting with 67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584 not found: ID does not exist" containerID="67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584" Nov 01 01:12:46.227569 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.227532 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584"} err="failed to get container status \"67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584\": rpc error: code = NotFound desc = could not find container \"67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584\": container with ID starting with 67b876e2629f5d5fc9a74f815fc33619b0c7964d3d1c7e247c3f0ceac2431584 not found: ID does not exist" Nov 01 01:12:46.382441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.382405 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq"] Nov 01 01:12:46.432874 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.432841 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/premium-w6-22d7b2-kserve-7bcd85cf79-mnctq"] Nov 01 01:12:46.893628 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:46.893594 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:47.178404 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.178318 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:47.249889 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.249856 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" path="/var/lib/kubelet/pods/af64c3fe-059e-4f14-9eb0-910b05dae0b5/volumes" Nov 01 01:12:47.400496 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.400467 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:47.550337 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.550304 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7"] Nov 01 01:12:47.550533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.550513 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" podUID="87a58ae1-5f3a-4556-a74d-5c3420fc172b" containerName="maas-api" containerID="cri-o://936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff" gracePeriod=30 Nov 01 01:12:47.720506 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.720469 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:47.767812 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.767777 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4"] Nov 01 01:12:47.768567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768548 6036 cpu_manager.go:414] "containerMap: removing container" podUID="675d3e25-4ab9-46be-878a-d049d7e50e58" containerName="payload-pre-processing" Nov 01 01:12:47.768630 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768571 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675d3e25-4ab9-46be-878a-d049d7e50e58" containerName="payload-pre-processing" Nov 01 01:12:47.768630 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768605 6036 cpu_manager.go:414] "containerMap: removing container" podUID="751ff7e8-5d91-4386-8882-c7d75d4e668f" containerName="payload-processing" Nov 01 01:12:47.768630 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768613 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751ff7e8-5d91-4386-8882-c7d75d4e668f" containerName="payload-processing" Nov 01 01:12:47.768739 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768637 6036 cpu_manager.go:414] "containerMap: removing container" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" Nov 01 01:12:47.768739 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768645 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" Nov 01 01:12:47.768739 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768662 6036 cpu_manager.go:414] "containerMap: removing container" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" Nov 01 01:12:47.768739 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768670 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" Nov 01 01:12:47.768876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768773 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="751ff7e8-5d91-4386-8882-c7d75d4e668f" containerName="payload-processing" Nov 01 01:12:47.768876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768787 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="af64c3fe-059e-4f14-9eb0-910b05dae0b5" containerName="main" Nov 01 01:12:47.768876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768799 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a0e24d-d19e-4750-9423-b6f61f034983" containerName="istio-proxy" Nov 01 01:12:47.768876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.768815 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="675d3e25-4ab9-46be-878a-d049d7e50e58" containerName="payload-pre-processing" Nov 01 01:12:47.780735 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.780703 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.844841 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.839488 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-48d289-openshift-default-dockercfg-bv5wf\"" Nov 01 01:12:47.844841 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.843265 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4"] Nov 01 01:12:47.854954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.854926 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 01:12:47.861502 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.861469 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a58ae1-5f3a-4556-a74d-5c3420fc172b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a58ae1-5f3a-4556-a74d-5c3420fc172b-openshift-service-ca\") pod \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " Nov 01 01:12:47.861806 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.861782 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls\") pod \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " Nov 01 01:12:47.862060 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862037 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-metrics-certs\") pod \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " Nov 01 01:12:47.862207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862096 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a58ae1-5f3a-4556-a74d-5c3420fc172b-kube-api-access-hpch9\" (UniqueName: \"kubernetes.io/projected/87a58ae1-5f3a-4556-a74d-5c3420fc172b-kube-api-access-hpch9\") pod \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\" (UID: \"87a58ae1-5f3a-4556-a74d-5c3420fc172b\") " Nov 01 01:12:47.862362 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862344 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-envoy\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.862493 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862409 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-credential-socket\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.862493 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862437 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdrb\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-kube-api-access-hgdrb\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.862682 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862500 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-data\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.862682 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862544 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-podinfo\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.862682 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862572 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istiod-ca-cert\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.862682 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862679 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-token\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.863255 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862736 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-certs\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.863255 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.862808 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-socket\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.863445 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.863254 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a58ae1-5f3a-4556-a74d-5c3420fc172b-openshift-service-ca" pod "87a58ae1-5f3a-4556-a74d-5c3420fc172b" (UID: "87a58ae1-5f3a-4556-a74d-5c3420fc172b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:47.867333 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.867276 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls" pod "87a58ae1-5f3a-4556-a74d-5c3420fc172b" (UID: "87a58ae1-5f3a-4556-a74d-5c3420fc172b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:47.868931 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.868889 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-metrics-certs" pod "87a58ae1-5f3a-4556-a74d-5c3420fc172b" (UID: "87a58ae1-5f3a-4556-a74d-5c3420fc172b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:47.871544 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.871520 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a58ae1-5f3a-4556-a74d-5c3420fc172b-kube-api-access-hpch9" pod "87a58ae1-5f3a-4556-a74d-5c3420fc172b" (UID: "87a58ae1-5f3a-4556-a74d-5c3420fc172b"). InnerVolumeSpecName "kube-api-access-hpch9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:47.959841 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.959804 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:47.963780 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.963723 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-certs\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.963990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.963788 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-socket\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.963990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.963831 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-envoy\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.963990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.963869 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-credential-socket\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.963990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.963890 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdrb\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-kube-api-access-hgdrb\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.963990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.963931 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-data\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.963990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.963958 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-podinfo\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.963990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.963980 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istiod-ca-cert\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.964415 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.964045 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-token\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.964415 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.964109 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:47.964415 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.964122 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a58ae1-5f3a-4556-a74d-5c3420fc172b-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:47.964415 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.964137 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpch9\" (UniqueName: \"kubernetes.io/projected/87a58ae1-5f3a-4556-a74d-5c3420fc172b-kube-api-access-hpch9\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:47.964415 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.964154 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a58ae1-5f3a-4556-a74d-5c3420fc172b-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:47.964681 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.964666 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-certs\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.964914 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.964893 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-socket\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.965181 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.965155 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-data\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.965319 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.965199 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-credential-socket\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.965563 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.965542 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istiod-ca-cert\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.968442 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.968413 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-envoy\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:47.968741 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:47.968717 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-podinfo\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:48.026193 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.026154 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-token\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:48.026918 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.026897 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdrb\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-kube-api-access-hgdrb\") pod \"e2e-inf-b-48d289-openshift-default-64669d6567-f52n4\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:48.153826 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.153753 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:48.208756 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.208708 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:48.231085 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.231048 6036 generic.go:356] "Generic (PLEG): container finished" podID="87a58ae1-5f3a-4556-a74d-5c3420fc172b" containerID="936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff" exitCode=0 Nov 01 01:12:48.231248 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.231104 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" event={"ID":"87a58ae1-5f3a-4556-a74d-5c3420fc172b","Type":"ContainerDied","Data":"936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff"} Nov 01 01:12:48.231248 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.231160 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" event={"ID":"87a58ae1-5f3a-4556-a74d-5c3420fc172b","Type":"ContainerDied","Data":"2a71e3ef22975daf6d71497203f83e4ea23947b4fa8f3f7050237b9c21f07710"} Nov 01 01:12:48.231248 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.231106 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7" Nov 01 01:12:48.231385 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.231192 6036 scope.go:122] "RemoveContainer" containerID="936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff" Nov 01 01:12:48.247878 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.247153 6036 scope.go:122] "RemoveContainer" containerID="936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff" Nov 01 01:12:48.247878 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:48.247662 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff\": container with ID starting with 936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff not found: ID does not exist" containerID="936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff" Nov 01 01:12:48.247878 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.247704 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff"} err="failed to get container status \"936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff\": rpc error: code = NotFound desc = could not find container \"936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff\": container with ID starting with 936da5cb8d718ff96872951759fbaf3863672ccf0d14f516a1435a759e34c1ff not found: ID does not exist" Nov 01 01:12:48.397364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.397317 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7"] Nov 01 01:12:48.401670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.401641 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-30fb19-76cbc8bd69-mnml7"] Nov 01 01:12:48.592499 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.592103 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4"] Nov 01 01:12:48.596065 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:12:48.596037 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec32ea80_f52f_4588_8e2b_3d42aa4ee258.slice/crio-225941ca4337bb5aed6b7728de893988218d6cc686fc5af71a97934753f93edd WatchSource:0}: Error finding container 225941ca4337bb5aed6b7728de893988218d6cc686fc5af71a97934753f93edd: Status 404 returned error can't find the container with id 225941ca4337bb5aed6b7728de893988218d6cc686fc5af71a97934753f93edd Nov 01 01:12:48.598815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.598774 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:48.598917 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.598865 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:48.598958 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:48.598932 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:49.237954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:49.237921 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" event={"ID":"ec32ea80-f52f-4588-8e2b-3d42aa4ee258","Type":"ContainerStarted","Data":"f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2"} Nov 01 01:12:49.238132 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:49.237971 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" event={"ID":"ec32ea80-f52f-4588-8e2b-3d42aa4ee258","Type":"ContainerStarted","Data":"225941ca4337bb5aed6b7728de893988218d6cc686fc5af71a97934753f93edd"} Nov 01 01:12:49.250640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:49.250604 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a58ae1-5f3a-4556-a74d-5c3420fc172b" path="/var/lib/kubelet/pods/87a58ae1-5f3a-4556-a74d-5c3420fc172b/volumes" Nov 01 01:12:49.361698 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:49.361638 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" podStartSLOduration=2.361622749 podStartE2EDuration="2.361622749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:49.358960378 +0000 UTC m=+2914.234181394" watchObservedRunningTime="2026-11-01 01:12:49.361622749 +0000 UTC m=+2914.236843763" Nov 01 01:12:49.370993 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:49.370962 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:49.517652 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:12:49.517573 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:12:49.694893 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:49.694861 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:49.938554 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:49.938521 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:50.154342 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:50.154307 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:50.160763 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:50.160732 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:50.245086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:50.244995 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:50.246134 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:50.246113 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:12:50.352144 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:50.352111 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:52.255623 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:52.255586 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:52.466641 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:52.466604 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:53.354172 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:53.354127 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:53.556432 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:53.556398 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:55.390476 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.390443 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:55.715068 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.714972 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:55.833407 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.833379 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj"] Nov 01 01:12:55.833873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.833860 6036 cpu_manager.go:414] "containerMap: removing container" podUID="87a58ae1-5f3a-4556-a74d-5c3420fc172b" containerName="maas-api" Nov 01 01:12:55.833934 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.833875 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a58ae1-5f3a-4556-a74d-5c3420fc172b" containerName="maas-api" Nov 01 01:12:55.833972 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.833940 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a58ae1-5f3a-4556-a74d-5c3420fc172b" containerName="maas-api" Nov 01 01:12:55.838238 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.838220 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:55.935892 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.935853 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:55.956588 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.954663 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-48d289-dockercfg-rrm4m\"" Nov 01 01:12:55.968606 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.968513 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-48d289\"" Nov 01 01:12:55.969130 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.969087 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj"] Nov 01 01:12:55.969268 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.969216 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj\" (UID: \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:55.969355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.969279 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whdsp\" (UniqueName: \"kubernetes.io/projected/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-kube-api-access-whdsp\") pod \"payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj\" (UID: \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:55.984928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.984894 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw"] Nov 01 01:12:55.988831 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:55.988804 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:56.046673 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.046643 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw"] Nov 01 01:12:56.070249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.070217 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15868285-24e8-4dfd-8700-9d20ec6f65c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw\" (UID: \"15868285-24e8-4dfd-8700-9d20ec6f65c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:56.070478 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.070438 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj\" (UID: \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:56.070558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.070478 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whdsp\" (UniqueName: \"kubernetes.io/projected/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-kube-api-access-whdsp\") pod \"payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj\" (UID: \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:56.070558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.070504 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxsdr\" (UniqueName: \"kubernetes.io/projected/15868285-24e8-4dfd-8700-9d20ec6f65c0-kube-api-access-sxsdr\") pod \"payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw\" (UID: \"15868285-24e8-4dfd-8700-9d20ec6f65c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:56.071138 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.071117 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj\" (UID: \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:56.132841 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.132814 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whdsp\" (UniqueName: \"kubernetes.io/projected/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-kube-api-access-whdsp\") pod \"payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj\" (UID: \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:56.151719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.151689 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:56.171689 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.171652 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxsdr\" (UniqueName: \"kubernetes.io/projected/15868285-24e8-4dfd-8700-9d20ec6f65c0-kube-api-access-sxsdr\") pod \"payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw\" (UID: \"15868285-24e8-4dfd-8700-9d20ec6f65c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:56.171833 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.171709 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15868285-24e8-4dfd-8700-9d20ec6f65c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw\" (UID: \"15868285-24e8-4dfd-8700-9d20ec6f65c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:56.172405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.172386 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15868285-24e8-4dfd-8700-9d20ec6f65c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw\" (UID: \"15868285-24e8-4dfd-8700-9d20ec6f65c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:56.233944 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.233864 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxsdr\" (UniqueName: \"kubernetes.io/projected/15868285-24e8-4dfd-8700-9d20ec6f65c0-kube-api-access-sxsdr\") pod \"payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw\" (UID: \"15868285-24e8-4dfd-8700-9d20ec6f65c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:56.301754 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.301726 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:56.530147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.530122 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj"] Nov 01 01:12:56.532144 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:12:56.532110 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb497c0_4e3b_46da_a860_c131fc8e7ec0.slice/crio-cb410ef953193a152eda6e48c1ed53e2ac3c02e461887226cb23f02d0662af81 WatchSource:0}: Error finding container cb410ef953193a152eda6e48c1ed53e2ac3c02e461887226cb23f02d0662af81: Status 404 returned error can't find the container with id cb410ef953193a152eda6e48c1ed53e2ac3c02e461887226cb23f02d0662af81 Nov 01 01:12:56.676562 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:56.676535 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw"] Nov 01 01:12:56.678428 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:12:56.678396 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15868285_24e8_4dfd_8700_9d20ec6f65c0.slice/crio-b31d3d95dd25809db486c6252c0e63e9916a3f2447b240346456a363aa45c239 WatchSource:0}: Error finding container b31d3d95dd25809db486c6252c0e63e9916a3f2447b240346456a363aa45c239: Status 404 returned error can't find the container with id b31d3d95dd25809db486c6252c0e63e9916a3f2447b240346456a363aa45c239 Nov 01 01:12:57.294437 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.294396 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" event={"ID":"15868285-24e8-4dfd-8700-9d20ec6f65c0","Type":"ContainerStarted","Data":"b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d"} Nov 01 01:12:57.294625 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.294450 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" event={"ID":"15868285-24e8-4dfd-8700-9d20ec6f65c0","Type":"ContainerStarted","Data":"b31d3d95dd25809db486c6252c0e63e9916a3f2447b240346456a363aa45c239"} Nov 01 01:12:57.294625 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.294499 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:12:57.296010 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.295982 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" event={"ID":"9cb497c0-4e3b-46da-a860-c131fc8e7ec0","Type":"ContainerStarted","Data":"d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201"} Nov 01 01:12:57.296101 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.296028 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" event={"ID":"9cb497c0-4e3b-46da-a860-c131fc8e7ec0","Type":"ContainerStarted","Data":"cb410ef953193a152eda6e48c1ed53e2ac3c02e461887226cb23f02d0662af81"} Nov 01 01:12:57.296101 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.296049 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:12:57.415458 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.415409 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" podStartSLOduration=2.415391883 podStartE2EDuration="2.415391883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:57.413231156 +0000 UTC m=+2922.288452172" watchObservedRunningTime="2026-11-01 01:12:57.415391883 +0000 UTC m=+2922.290612898" Nov 01 01:12:57.531689 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.531638 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" podStartSLOduration=2.531617201 podStartE2EDuration="2.531617201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:57.530234282 +0000 UTC m=+2922.405455297" watchObservedRunningTime="2026-11-01 01:12:57.531617201 +0000 UTC m=+2922.406838207" Nov 01 01:12:57.682679 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.682567 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:57.889739 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.889708 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:12:57.967477 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.967398 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk"] Nov 01 01:12:57.967648 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:57.967604 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" containerID="cri-o://3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746" gracePeriod=30 Nov 01 01:12:58.101349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:12:58.101302 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:00.263132 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:00.263101 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:03.226808 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.226780 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 01:13:03.327978 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.327941 6036 generic.go:356] "Generic (PLEG): container finished" podID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerID="3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746" exitCode=0 Nov 01 01:13:03.328187 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.328000 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" Nov 01 01:13:03.328187 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.328029 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" event={"ID":"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0","Type":"ContainerDied","Data":"3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746"} Nov 01 01:13:03.328187 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.328084 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk" event={"ID":"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0","Type":"ContainerDied","Data":"2fbd30e75ec0d95fb3bb927ae006525e130fff78f1a07e93577830dac886b6cb"} Nov 01 01:13:03.328187 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.328108 6036 scope.go:122] "RemoveContainer" containerID="3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746" Nov 01 01:13:03.337514 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.337499 6036 scope.go:122] "RemoveContainer" containerID="3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746" Nov 01 01:13:03.337778 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:13:03.337760 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746\": container with ID starting with 3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746 not found: ID does not exist" containerID="3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746" Nov 01 01:13:03.337829 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.337785 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746"} err="failed to get container status \"3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746\": rpc error: code = NotFound desc = could not find container \"3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746\": container with ID starting with 3810d6bf97b973e636e778d88b8f396d45a8cae674b5ffbb729e261e27a8a746 not found: ID does not exist" Nov 01 01:13:03.354318 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354271 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-envoy\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.354420 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354403 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-credential-socket\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.354486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354457 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-data\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.354547 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354495 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istiod-ca-cert\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.354547 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354530 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-kube-api-access-dmtmh\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-kube-api-access-dmtmh\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.354692 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354665 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-certs\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.354837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354705 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-credential-socket" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:03.354837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354735 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-podinfo\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.354837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354768 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-token\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-token\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.354837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354795 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-data" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:03.354837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354813 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-socket\") pod \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\" (UID: \"c3d625cd-d8fc-41ea-a34c-9400bbf84dd0\") " Nov 01 01:13:03.355119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354937 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-certs" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:03.355119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.354975 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istiod-ca-cert" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:13:03.355339 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.355318 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-socket" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:03.355432 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.355326 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.355432 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.355364 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.355432 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.355382 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.355432 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.355397 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.356704 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.356682 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-envoy" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:03.357204 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.357184 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-token" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:13:03.357375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.357355 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-podinfo" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:13:03.357447 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.357376 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-kube-api-access-dmtmh" pod "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" (UID: "c3d625cd-d8fc-41ea-a34c-9400bbf84dd0"). InnerVolumeSpecName "kube-api-access-dmtmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:13:03.456299 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.456257 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmtmh\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-kube-api-access-dmtmh\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.456460 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.456312 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.456460 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.456328 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.456460 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.456342 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.456460 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.456355 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:03.750038 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.749999 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk"] Nov 01 01:13:03.755968 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:03.755929 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-skzsk"] Nov 01 01:13:05.159444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:05.159410 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:05.249932 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:05.249900 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" path="/var/lib/kubelet/pods/c3d625cd-d8fc-41ea-a34c-9400bbf84dd0/volumes" Nov 01 01:13:07.679272 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:07.679219 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:07.899731 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:07.899698 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:08.305110 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:08.305080 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:13:08.305327 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:08.305153 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:13:08.692740 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:08.692662 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:09.647914 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.647873 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf"] Nov 01 01:13:09.648512 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.648492 6036 cpu_manager.go:414] "containerMap: removing container" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" Nov 01 01:13:09.648512 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.648510 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" Nov 01 01:13:09.648664 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.648620 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d625cd-d8fc-41ea-a34c-9400bbf84dd0" containerName="istio-proxy" Nov 01 01:13:09.653331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.653313 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.708583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.708542 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-48d289-serving-cert\"" Nov 01 01:13:09.731100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.731064 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.731323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.731192 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80eca316-5f74-41f7-9ac7-2ca9c73913ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.731323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.731249 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876mq\" (UniqueName: \"kubernetes.io/projected/80eca316-5f74-41f7-9ac7-2ca9c73913ff-kube-api-access-876mq\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.731323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.731279 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-metrics-certs\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.761063 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.761033 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf"] Nov 01 01:13:09.832441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.832409 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-metrics-certs\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.832638 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.832490 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.832638 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.832553 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80eca316-5f74-41f7-9ac7-2ca9c73913ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.832638 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.832589 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876mq\" (UniqueName: \"kubernetes.io/projected/80eca316-5f74-41f7-9ac7-2ca9c73913ff-kube-api-access-876mq\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.832828 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:13:09.832638 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-serving-cert: secret "maas-api-e2e-inf-b-48d289-serving-cert" not found Nov 01 01:13:09.832828 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:13:09.832715 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls podName:80eca316-5f74-41f7-9ac7-2ca9c73913ff nodeName:}" failed. No retries permitted until 2026-11-01 01:13:10.332693745 +0000 UTC m=+2935.207914757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls") pod "maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" (UID: "80eca316-5f74-41f7-9ac7-2ca9c73913ff") : secret "maas-api-e2e-inf-b-48d289-serving-cert" not found Nov 01 01:13:09.833192 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.833173 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80eca316-5f74-41f7-9ac7-2ca9c73913ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.835105 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.835084 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-metrics-certs\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.896215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.896177 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-876mq\" (UniqueName: \"kubernetes.io/projected/80eca316-5f74-41f7-9ac7-2ca9c73913ff-kube-api-access-876mq\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:09.909074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:09.908957 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:10.124073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:10.124042 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:10.331429 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:10.331397 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:10.338867 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:10.338837 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:10.341554 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:10.341533 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:10.566526 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:10.566489 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:10.945611 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:10.945584 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf"] Nov 01 01:13:10.948017 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:13:10.947987 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80eca316_5f74_41f7_9ac7_2ca9c73913ff.slice/crio-393314e757ab29fad4c502ce47dfa2dc362b50313059ea06622f7eeb17081f0a WatchSource:0}: Error finding container 393314e757ab29fad4c502ce47dfa2dc362b50313059ea06622f7eeb17081f0a: Status 404 returned error can't find the container with id 393314e757ab29fad4c502ce47dfa2dc362b50313059ea06622f7eeb17081f0a Nov 01 01:13:10.949671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:10.949646 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 01:13:11.374132 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:11.374093 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" event={"ID":"80eca316-5f74-41f7-9ac7-2ca9c73913ff","Type":"ContainerStarted","Data":"393314e757ab29fad4c502ce47dfa2dc362b50313059ea06622f7eeb17081f0a"} Nov 01 01:13:12.383148 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:12.383114 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" event={"ID":"80eca316-5f74-41f7-9ac7-2ca9c73913ff","Type":"ContainerStarted","Data":"9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf"} Nov 01 01:13:12.383666 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:12.383221 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:12.503048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:12.502987 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" podStartSLOduration=2.682728063 podStartE2EDuration="3.502966879s" totalImagesPullingTime="820.238816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:13:12.502372742 +0000 UTC m=+2937.377593759" watchObservedRunningTime="2026-11-01 01:13:12.502966879 +0000 UTC m=+2937.378187894" Nov 01 01:13:14.978067 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:14.978024 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:15.343684 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:15.343641 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:18.394782 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:18.394753 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:13:19.283118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:19.283085 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:19.524191 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:13:19.524147 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:13:20.808352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:20.808316 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:21.089918 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:21.089823 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:21.318686 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:21.318653 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:21.539173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:21.539141 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:21.760699 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:21.760663 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:26.157206 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:26.157169 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:28.067073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:28.067030 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:28.314462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:28.314421 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:28.542446 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:28.542412 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:31.056803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:31.056764 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:35.939659 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:35.939611 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:36.287482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:36.287447 6036 patch_prober.go:29] interesting pod/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 01:13:36.287667 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:36.287520 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 01:13:37.463342 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:37.463273 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:37.709541 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:37.709501 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:38.533717 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:38.533688 6036 generic.go:356] "Generic (PLEG): container finished" podID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerID="f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2" exitCode=0 Nov 01 01:13:38.534158 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:38.533723 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" event={"ID":"ec32ea80-f52f-4588-8e2b-3d42aa4ee258","Type":"ContainerDied","Data":"f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2"} Nov 01 01:13:38.534158 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:38.534061 6036 scope.go:122] "RemoveContainer" containerID="f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2" Nov 01 01:13:38.535394 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:38.535363 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:13:38.535474 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:38.535436 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:13:38.535474 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:38.535469 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:13:39.540933 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:39.540897 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" event={"ID":"ec32ea80-f52f-4588-8e2b-3d42aa4ee258","Type":"ContainerStarted","Data":"73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702"} Nov 01 01:13:40.156061 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:40.156023 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:13:40.156523 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:40.156485 6036 patch_prober.go:29] interesting pod/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 01:13:40.156722 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:40.156695 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 01:13:41.128164 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:41.128132 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:41.155716 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:41.155678 6036 patch_prober.go:29] interesting pod/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 01:13:41.156035 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:41.156005 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 01:13:42.159311 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:42.158700 6036 patch_prober.go:29] interesting pod/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 01:13:42.159311 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:42.158775 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 01:13:43.165313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:43.163732 6036 patch_prober.go:29] interesting pod/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 01:13:43.165313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:43.163796 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 01:13:44.161778 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:44.161740 6036 patch_prober.go:29] interesting pod/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 01:13:44.162088 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:44.162056 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 01:13:44.386141 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:44.386073 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:45.158558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:45.158529 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:13:45.158915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:45.158893 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:13:45.159592 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:45.159570 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:13:45.944219 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:45.944185 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:49.505855 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:13:49.505822 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:13:50.877714 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:50.877676 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:13:55.644784 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:13:55.644692 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:00.448683 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:00.448646 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:05.337249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:05.337214 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:08.325529 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:08.325495 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:08.538316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:08.538255 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:09.816016 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:09.815966 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:11.580127 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:11.580089 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:15.044415 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:15.044377 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:18.288229 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:18.288194 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:19.502993 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:14:19.502956 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:14:19.933583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:19.933547 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:24.748862 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:24.748821 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:29.647531 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:29.647494 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:34.610317 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:34.610254 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:37.894957 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:37.894917 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:39.442486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:39.442450 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:44.273052 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:44.273017 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:49.145333 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:49.145242 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:49.504684 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:14:49.504601 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:14:53.988566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:53.988519 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:58.414847 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:58.414804 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:58.627195 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:58.627160 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:14:58.835217 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:14:58.835171 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:03.676994 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:03.676959 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:08.665888 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:08.665848 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:13.550790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:13.550743 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:14.994413 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:14.994372 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:18.356104 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:18.356067 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:19.507858 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:15:19.507824 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:15:23.157494 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:23.157454 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:28.032192 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:28.032098 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:28.348646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:28.348608 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:28.560643 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:28.560603 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:32.948755 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:32.948712 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:37.747945 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:37.747905 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:38.162016 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:38.161979 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:42.551015 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:42.550983 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:47.441873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:47.441839 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:49.503203 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:15:49.503162 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:15:52.329725 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:52.329687 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:15:57.042882 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:15:57.042838 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:01.865686 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:01.865646 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:06.750214 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:06.750177 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:11.557364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:11.557323 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:16.361893 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:16.361856 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:19.507628 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:16:19.507585 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:16:19.558727 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:19.558690 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:21.159026 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:21.158989 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:22.229023 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:22.228985 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb"] Nov 01 01:16:22.229602 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:22.229250 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" containerID="cri-o://2d2d629ccc961d7975a684861b8827442fd31344270c73c09cea188cd66f23ea" gracePeriod=30 Nov 01 01:16:22.279904 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:22.279875 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm"] Nov 01 01:16:22.280119 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:22.280096 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" containerID="cri-o://a86da17d206b1742d78e7bf96d5bc26dda443b7011cb80f69bd854e4313c9fe0" gracePeriod=30 Nov 01 01:16:22.281556 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:22.281532 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn"] Nov 01 01:16:22.281748 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:22.281707 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" containerID="cri-o://f3832572f819b903cde7180fd971b74fc592e028caeac4984a0de2f029efa98b" gracePeriod=30 Nov 01 01:16:22.330449 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:22.330413 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt"] Nov 01 01:16:22.330679 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:22.330652 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" containerID="cri-o://0b94feb6a1f7bf9b4d1500c8bc2bb35744bff3af84c5ae182e3fb11a410f68dc" gracePeriod=30 Nov 01 01:16:26.039440 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:26.039394 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:27.497392 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.497048 6036 generic.go:356] "Generic (PLEG): container finished" podID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerID="a86da17d206b1742d78e7bf96d5bc26dda443b7011cb80f69bd854e4313c9fe0" exitCode=0 Nov 01 01:16:27.497392 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.497236 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" event={"ID":"caec49be-2da6-47ae-aad3-9ec0a797c6f8","Type":"ContainerDied","Data":"a86da17d206b1742d78e7bf96d5bc26dda443b7011cb80f69bd854e4313c9fe0"} Nov 01 01:16:27.501630 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.501416 6036 generic.go:356] "Generic (PLEG): container finished" podID="da902477-5e58-4752-b788-4488ce5821c5" containerID="f3832572f819b903cde7180fd971b74fc592e028caeac4984a0de2f029efa98b" exitCode=0 Nov 01 01:16:27.501630 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.501544 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" event={"ID":"da902477-5e58-4752-b788-4488ce5821c5","Type":"ContainerDied","Data":"f3832572f819b903cde7180fd971b74fc592e028caeac4984a0de2f029efa98b"} Nov 01 01:16:27.504592 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.504177 6036 generic.go:356] "Generic (PLEG): container finished" podID="cde2721d-f699-4371-b0e8-584978a0df59" containerID="2d2d629ccc961d7975a684861b8827442fd31344270c73c09cea188cd66f23ea" exitCode=0 Nov 01 01:16:27.504592 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.504253 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" event={"ID":"cde2721d-f699-4371-b0e8-584978a0df59","Type":"ContainerDied","Data":"2d2d629ccc961d7975a684861b8827442fd31344270c73c09cea188cd66f23ea"} Nov 01 01:16:27.506749 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.506648 6036 generic.go:356] "Generic (PLEG): container finished" podID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerID="0b94feb6a1f7bf9b4d1500c8bc2bb35744bff3af84c5ae182e3fb11a410f68dc" exitCode=0 Nov 01 01:16:27.506749 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.506681 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" event={"ID":"95829749-19bd-4ac0-86e8-a3635a805ea7","Type":"ContainerDied","Data":"0b94feb6a1f7bf9b4d1500c8bc2bb35744bff3af84c5ae182e3fb11a410f68dc"} Nov 01 01:16:27.541402 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.541371 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:16:27.615744 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.615701 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 01:16:27.654766 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.654740 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 01:16:27.657814 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.657792 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:16:27.688977 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.688945 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-socket\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689128 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689002 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-istio-token\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-istio-token\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689128 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689043 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-credential-socket\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689128 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689069 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-certs\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689128 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689097 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-envoy\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689168 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-data\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689205 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cde2721d-f699-4371-b0e8-584978a0df59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde2721d-f699-4371-b0e8-584978a0df59-istio-podinfo\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689221 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-socket" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.689387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689273 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde2721d-f699-4371-b0e8-584978a0df59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde2721d-f699-4371-b0e8-584978a0df59-istiod-ca-cert\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689421 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-credential-socket" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.689639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689427 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-kube-api-access-62kbw\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-kube-api-access-62kbw\") pod \"cde2721d-f699-4371-b0e8-584978a0df59\" (UID: \"cde2721d-f699-4371-b0e8-584978a0df59\") " Nov 01 01:16:27.689639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689435 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-certs" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.689639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689573 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-data" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.689850 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.689665 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde2721d-f699-4371-b0e8-584978a0df59-istiod-ca-cert" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:16:27.690201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.690174 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.690201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.690204 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.690201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.690220 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.690201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.690234 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.690201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.690248 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde2721d-f699-4371-b0e8-584978a0df59-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.691621 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.691594 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-envoy" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.691621 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.691609 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde2721d-f699-4371-b0e8-584978a0df59-istio-podinfo" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:16:27.691770 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.691655 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-kube-api-access-62kbw" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "kube-api-access-62kbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:27.691883 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.691866 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-istio-token" pod "cde2721d-f699-4371-b0e8-584978a0df59" (UID: "cde2721d-f699-4371-b0e8-584978a0df59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:27.791394 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791360 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da902477-5e58-4752-b788-4488ce5821c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da902477-5e58-4752-b788-4488ce5821c5-istiod-ca-cert\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.791394 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791399 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-envoy\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.791665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791418 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-socket\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.791665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791436 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-credential-socket\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.791665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791471 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-credential-socket\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.791665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791491 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-data\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.791665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791514 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-kube-api-access-gvbms\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-kube-api-access-gvbms\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.791665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791547 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-podinfo\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.791665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791661 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-socket\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.792040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791733 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-socket" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.792040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791796 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-data" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.792040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791855 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-credential-socket" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.792040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791805 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-credential-socket" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.792040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791907 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da902477-5e58-4752-b788-4488ce5821c5-istiod-ca-cert" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:16:27.792040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791940 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-token\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-token\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.792040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.791974 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-socket\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792052 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-kube-api-access-tqbkt\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-kube-api-access-tqbkt\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792121 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95829749-19bd-4ac0-86e8-a3635a805ea7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95829749-19bd-4ac0-86e8-a3635a805ea7-istiod-ca-cert\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792151 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-credential-socket\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792191 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-kube-api-access-mhtg2\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-kube-api-access-mhtg2\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792221 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-envoy\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792253 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-socket" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792258 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-token\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-token\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792321 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-data\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792342 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-socket" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792379 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-podinfo\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792409 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-certs\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.792452 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792436 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-certs\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792469 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-certs\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792522 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-istio-token\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-istio-token\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792555 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da902477-5e58-4752-b788-4488ce5821c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da902477-5e58-4752-b788-4488ce5821c5-istio-podinfo\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792580 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-data\") pod \"da902477-5e58-4752-b788-4488ce5821c5\" (UID: \"da902477-5e58-4752-b788-4488ce5821c5\") " Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792626 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-envoy\") pod \"95829749-19bd-4ac0-86e8-a3635a805ea7\" (UID: \"95829749-19bd-4ac0-86e8-a3635a805ea7\") " Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792657 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istiod-ca-cert\") pod \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\" (UID: \"caec49be-2da6-47ae-aad3-9ec0a797c6f8\") " Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792833 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-certs" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.792996 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-credential-socket" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.793125 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-data" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.793364 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.793154 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istiod-ca-cert" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:16:27.793907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.793618 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-data" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.794177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794147 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-certs" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.794322 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794198 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95829749-19bd-4ac0-86e8-a3635a805ea7-istiod-ca-cert" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:16:27.794322 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794315 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-certs" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.794462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794395 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794432 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794444 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794460 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da902477-5e58-4752-b788-4488ce5821c5-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794476 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794490 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794504 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794517 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794530 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794545 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde2721d-f699-4371-b0e8-584978a0df59-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794558 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794574 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde2721d-f699-4371-b0e8-584978a0df59-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794587 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794603 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62kbw\" (UniqueName: \"kubernetes.io/projected/cde2721d-f699-4371-b0e8-584978a0df59-kube-api-access-62kbw\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.794694 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794617 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.795398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.794719 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-kube-api-access-gvbms" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "kube-api-access-gvbms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:27.795891 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.795864 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-kube-api-access-tqbkt" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "kube-api-access-tqbkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:27.796077 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.796054 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-kube-api-access-mhtg2" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "kube-api-access-mhtg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:27.796154 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.796118 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-envoy" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.796343 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.796270 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-token" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:27.796343 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.796309 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-envoy" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.796569 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.796546 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-podinfo" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:16:27.796654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.796635 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-envoy" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:16:27.797089 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.797071 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-token" pod "95829749-19bd-4ac0-86e8-a3635a805ea7" (UID: "95829749-19bd-4ac0-86e8-a3635a805ea7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:27.797258 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.797243 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-istio-token" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:27.797834 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.797819 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da902477-5e58-4752-b788-4488ce5821c5-istio-podinfo" pod "da902477-5e58-4752-b788-4488ce5821c5" (UID: "da902477-5e58-4752-b788-4488ce5821c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:16:27.797920 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.797906 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-podinfo" pod "caec49be-2da6-47ae-aad3-9ec0a797c6f8" (UID: "caec49be-2da6-47ae-aad3-9ec0a797c6f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:16:27.896398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896369 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896395 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896405 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896414 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896423 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896434 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da902477-5e58-4752-b788-4488ce5821c5-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896443 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896451 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896459 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da902477-5e58-4752-b788-4488ce5821c5-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896467 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvbms\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-kube-api-access-gvbms\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896476 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95829749-19bd-4ac0-86e8-a3635a805ea7-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896485 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896494 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbkt\" (UniqueName: \"kubernetes.io/projected/95829749-19bd-4ac0-86e8-a3635a805ea7-kube-api-access-tqbkt\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896504 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95829749-19bd-4ac0-86e8-a3635a805ea7-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896516 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtg2\" (UniqueName: \"kubernetes.io/projected/da902477-5e58-4752-b788-4488ce5821c5-kube-api-access-mhtg2\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:27.896678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:27.896524 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caec49be-2da6-47ae-aad3-9ec0a797c6f8-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:28.513261 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.513181 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" Nov 01 01:16:28.513762 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.513183 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn" event={"ID":"da902477-5e58-4752-b788-4488ce5821c5","Type":"ContainerDied","Data":"6bddd55ab7c7b45ee2b3da32feff97b778814e2d02709e6d4e5832cb7dd53258"} Nov 01 01:16:28.513762 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.513347 6036 scope.go:122] "RemoveContainer" containerID="f3832572f819b903cde7180fd971b74fc592e028caeac4984a0de2f029efa98b" Nov 01 01:16:28.514769 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.514751 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" Nov 01 01:16:28.514879 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.514751 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb" event={"ID":"cde2721d-f699-4371-b0e8-584978a0df59","Type":"ContainerDied","Data":"84654cd77e8aebfc13841f9c6b4bfc90c7264ef7daaee69a708a50be531a812e"} Nov 01 01:16:28.516358 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.516337 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" Nov 01 01:16:28.516469 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.516378 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt" event={"ID":"95829749-19bd-4ac0-86e8-a3635a805ea7","Type":"ContainerDied","Data":"1b1bcf2500b99a3c5c9e0335b3fdf8df631691dd4d2666edd59b2a287e2bf21d"} Nov 01 01:16:28.518116 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.518037 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" event={"ID":"caec49be-2da6-47ae-aad3-9ec0a797c6f8","Type":"ContainerDied","Data":"9155cbb046420f6990462dd914bbd4c5fbd2442a81859dc5dcb79cab0523281d"} Nov 01 01:16:28.518116 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.518061 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm" Nov 01 01:16:28.525446 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.525426 6036 scope.go:122] "RemoveContainer" containerID="2d2d629ccc961d7975a684861b8827442fd31344270c73c09cea188cd66f23ea" Nov 01 01:16:28.535566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.535551 6036 scope.go:122] "RemoveContainer" containerID="0b94feb6a1f7bf9b4d1500c8bc2bb35744bff3af84c5ae182e3fb11a410f68dc" Nov 01 01:16:28.544062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.544047 6036 scope.go:122] "RemoveContainer" containerID="a86da17d206b1742d78e7bf96d5bc26dda443b7011cb80f69bd854e4313c9fe0" Nov 01 01:16:28.681009 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.680975 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm"] Nov 01 01:16:28.684811 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.684782 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-94qbm"] Nov 01 01:16:28.795743 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.795713 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt"] Nov 01 01:16:28.801440 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.801413 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-swvvt"] Nov 01 01:16:28.912657 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.912622 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn"] Nov 01 01:16:28.917907 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:28.917881 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dd6kn"] Nov 01 01:16:29.028618 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:29.028583 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb"] Nov 01 01:16:29.033008 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:29.032981 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gbrgb"] Nov 01 01:16:29.250478 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:29.250404 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" path="/var/lib/kubelet/pods/95829749-19bd-4ac0-86e8-a3635a805ea7/volumes" Nov 01 01:16:29.250866 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:29.250854 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" path="/var/lib/kubelet/pods/caec49be-2da6-47ae-aad3-9ec0a797c6f8/volumes" Nov 01 01:16:29.251250 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:29.251237 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde2721d-f699-4371-b0e8-584978a0df59" path="/var/lib/kubelet/pods/cde2721d-f699-4371-b0e8-584978a0df59/volumes" Nov 01 01:16:29.251963 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:29.251948 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da902477-5e58-4752-b788-4488ce5821c5" path="/var/lib/kubelet/pods/da902477-5e58-4752-b788-4488ce5821c5/volumes" Nov 01 01:16:30.847107 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:30.847070 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:35.647710 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:35.647655 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:38.526637 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:38.526589 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:40.441061 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:40.441024 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:45.340760 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:45.340729 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:49.504791 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:16:49.504755 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:16:50.153101 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:50.153067 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:56.568566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:56.568535 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:16:59.787891 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:16:59.787855 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:01.898380 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:01.898336 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:04.669097 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:04.669060 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:09.553974 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:09.553937 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:14.351735 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:14.351706 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:19.247782 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:19.247745 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:19.504671 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:17:19.504585 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:17:24.146433 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:24.146403 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:29.039706 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:29.039677 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:30.800609 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:30.800580 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:34.241584 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:34.241548 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:38.874914 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:38.874883 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:39.249558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:39.249470 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:43.222272 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:43.222235 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:43.434031 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:43.433994 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:44.126708 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:44.126674 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:48.410230 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:48.410192 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:49.504827 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:17:49.504783 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:17:50.009684 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:50.009648 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:50.245555 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:50.245517 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:50.458483 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:50.458450 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:51.150423 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:51.150388 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:55.453324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:55.453265 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:57.472363 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:57.472323 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:57.698972 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:57.698939 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:57.727174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:57.727096 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z"] Nov 01 01:17:57.727369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:57.727331 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" containerID="cri-o://8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca" gracePeriod=30 Nov 01 01:17:57.908497 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:57.908450 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:17:59.266151 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:59.266114 6036 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:17:59.266572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:17:59.266170 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:18:00.581963 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:00.581929 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:02.979237 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:02.979206 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:18:03.011141 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011067 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-token\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-token\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011141 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011109 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-socket\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011167 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-kube-api-access-n8qp9\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-kube-api-access-n8qp9\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011190 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-data\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011215 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-certs\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011252 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-podinfo\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011279 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istiod-ca-cert\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011386 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011326 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-credential-socket\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011706 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011497 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-envoy\") pod \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\" (UID: \"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47\") " Nov 01 01:18:03.011706 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011536 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-data" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.011706 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011555 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-certs" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.011706 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011563 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-socket" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.011939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011778 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-credential-socket" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.011939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011879 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.011939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011898 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.011939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011905 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istiod-ca-cert" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:03.011939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011914 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.012402 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.011970 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.013756 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.013730 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-kube-api-access-n8qp9" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "kube-api-access-n8qp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:03.013896 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.013876 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-envoy" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.013959 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.013911 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-podinfo" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:18:03.014022 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.013996 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-token" pod "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" (UID: "adf52ec0-ebbf-44ac-bc3e-dae9caac2d47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:03.016025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.015998 6036 generic.go:356] "Generic (PLEG): container finished" podID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerID="8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca" exitCode=0 Nov 01 01:18:03.016115 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.016057 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" Nov 01 01:18:03.016115 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.016083 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" event={"ID":"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47","Type":"ContainerDied","Data":"8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca"} Nov 01 01:18:03.016192 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.016162 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z" event={"ID":"adf52ec0-ebbf-44ac-bc3e-dae9caac2d47","Type":"ContainerDied","Data":"302ff0529c6fe4bfbf442334efa00d477afd79a5fcb3e883062a18ed4c5f5b66"} Nov 01 01:18:03.016192 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.016188 6036 scope.go:122] "RemoveContainer" containerID="8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca" Nov 01 01:18:03.035947 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.035927 6036 scope.go:122] "RemoveContainer" containerID="8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca" Nov 01 01:18:03.036238 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:03.036220 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca\": container with ID starting with 8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca not found: ID does not exist" containerID="8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca" Nov 01 01:18:03.036322 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.036252 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca"} err="failed to get container status \"8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca\": rpc error: code = NotFound desc = could not find container \"8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca\": container with ID starting with 8ea200aec29d565799beb5f75af6bdd19e209a006f19159081baad52f28d9cca not found: ID does not exist" Nov 01 01:18:03.113563 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.113529 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.113563 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.113558 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.113563 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.113569 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8qp9\" (UniqueName: \"kubernetes.io/projected/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-kube-api-access-n8qp9\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.113808 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.113580 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.113808 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.113589 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.267757 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.267678 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z"] Nov 01 01:18:03.274563 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:03.274533 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-nhl7z"] Nov 01 01:18:05.249262 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:05.249232 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" path="/var/lib/kubelet/pods/adf52ec0-ebbf-44ac-bc3e-dae9caac2d47/volumes" Nov 01 01:18:05.760546 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:05.760512 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:10.109247 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:10.109211 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw"] Nov 01 01:18:10.215717 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:10.215679 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj"] Nov 01 01:18:10.583873 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:10.583841 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:10.800098 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:10.800062 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:14.678086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:14.678056 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf"] Nov 01 01:18:14.678548 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:14.678323 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" podUID="80eca316-5f74-41f7-9ac7-2ca9c73913ff" containerName="maas-api" containerID="cri-o://9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf" gracePeriod=30 Nov 01 01:18:14.842989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:14.842955 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:14.940962 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:14.940895 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:18:15.033326 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.033270 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-metrics-certs\") pod \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " Nov 01 01:18:15.033539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.033389 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls\") pod \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " Nov 01 01:18:15.033539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.033524 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80eca316-5f74-41f7-9ac7-2ca9c73913ff-kube-api-access-876mq\" (UniqueName: \"kubernetes.io/projected/80eca316-5f74-41f7-9ac7-2ca9c73913ff-kube-api-access-876mq\") pod \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " Nov 01 01:18:15.033670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.033565 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80eca316-5f74-41f7-9ac7-2ca9c73913ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80eca316-5f74-41f7-9ac7-2ca9c73913ff-openshift-service-ca\") pod \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\" (UID: \"80eca316-5f74-41f7-9ac7-2ca9c73913ff\") " Nov 01 01:18:15.033962 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.033935 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80eca316-5f74-41f7-9ac7-2ca9c73913ff-openshift-service-ca" pod "80eca316-5f74-41f7-9ac7-2ca9c73913ff" (UID: "80eca316-5f74-41f7-9ac7-2ca9c73913ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:15.035749 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.035725 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-metrics-certs" pod "80eca316-5f74-41f7-9ac7-2ca9c73913ff" (UID: "80eca316-5f74-41f7-9ac7-2ca9c73913ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:18:15.035749 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.035727 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80eca316-5f74-41f7-9ac7-2ca9c73913ff-kube-api-access-876mq" pod "80eca316-5f74-41f7-9ac7-2ca9c73913ff" (UID: "80eca316-5f74-41f7-9ac7-2ca9c73913ff"). InnerVolumeSpecName "kube-api-access-876mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:15.035913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.035745 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls" pod "80eca316-5f74-41f7-9ac7-2ca9c73913ff" (UID: "80eca316-5f74-41f7-9ac7-2ca9c73913ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:18:15.079816 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.079782 6036 generic.go:356] "Generic (PLEG): container finished" podID="80eca316-5f74-41f7-9ac7-2ca9c73913ff" containerID="9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf" exitCode=0 Nov 01 01:18:15.079993 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.079863 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" Nov 01 01:18:15.079993 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.079865 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" event={"ID":"80eca316-5f74-41f7-9ac7-2ca9c73913ff","Type":"ContainerDied","Data":"9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf"} Nov 01 01:18:15.079993 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.079986 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf" event={"ID":"80eca316-5f74-41f7-9ac7-2ca9c73913ff","Type":"ContainerDied","Data":"393314e757ab29fad4c502ce47dfa2dc362b50313059ea06622f7eeb17081f0a"} Nov 01 01:18:15.080124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.080008 6036 scope.go:122] "RemoveContainer" containerID="9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf" Nov 01 01:18:15.091994 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.090205 6036 scope.go:122] "RemoveContainer" containerID="9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf" Nov 01 01:18:15.091994 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:15.090639 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf\": container with ID starting with 9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf not found: ID does not exist" containerID="9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf" Nov 01 01:18:15.091994 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.090677 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf"} err="failed to get container status \"9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf\": rpc error: code = NotFound desc = could not find container \"9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf\": container with ID starting with 9c2725c414cf7d1bf5b29867f2ed63b983c8da0f8b8d9c839a1bd504c6c07baf not found: ID does not exist" Nov 01 01:18:15.110533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.110491 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" podUID="15868285-24e8-4dfd-8700-9d20ec6f65c0" containerName="payload-processing" containerID="cri-o://b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d" gracePeriod=25 Nov 01 01:18:15.135014 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.134973 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80eca316-5f74-41f7-9ac7-2ca9c73913ff-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:15.135014 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.135011 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:15.135014 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.135021 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80eca316-5f74-41f7-9ac7-2ca9c73913ff-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:15.135245 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.135029 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-876mq\" (UniqueName: \"kubernetes.io/projected/80eca316-5f74-41f7-9ac7-2ca9c73913ff-kube-api-access-876mq\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:15.216156 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.216064 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" podUID="9cb497c0-4e3b-46da-a860-c131fc8e7ec0" containerName="payload-pre-processing" containerID="cri-o://d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201" gracePeriod=25 Nov 01 01:18:15.251507 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.251470 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf"] Nov 01 01:18:15.258147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:15.258117 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d289-78956bcd4b-l7fwf"] Nov 01 01:18:16.416083 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:16.416051 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:16.666917 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:16.666835 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:16.885151 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:16.885084 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:17.249297 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:17.249248 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80eca316-5f74-41f7-9ac7-2ca9c73913ff" path="/var/lib/kubelet/pods/80eca316-5f74-41f7-9ac7-2ca9c73913ff/volumes" Nov 01 01:18:17.404095 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:17.404059 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:18.303052 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:18.303012 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" podUID="9cb497c0-4e3b-46da-a860-c131fc8e7ec0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:18:18.303527 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:18.303020 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" podUID="15868285-24e8-4dfd-8700-9d20ec6f65c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:18:19.503575 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:19.503530 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:18:21.843412 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:21.843371 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:23.840130 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:23.840099 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:24.061572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:24.061483 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:24.157417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:24.157334 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4"] Nov 01 01:18:24.157581 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:24.157539 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" containerID="cri-o://73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702" gracePeriod=30 Nov 01 01:18:24.368876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:24.368834 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:24.724302 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:24.724252 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:26.970396 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:26.970364 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:28.303042 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:28.302997 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" podUID="15868285-24e8-4dfd-8700-9d20ec6f65c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:18:28.303042 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:28.302998 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" podUID="9cb497c0-4e3b-46da-a860-c131fc8e7ec0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:18:29.408430 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.408404 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:18:29.485297 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485196 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-podinfo\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485297 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485243 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istiod-ca-cert\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485297 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485273 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-token\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-token\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485345 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-envoy\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485384 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-data\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485442 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-credential-socket\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485523 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-kube-api-access-hgdrb\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-kube-api-access-hgdrb\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485573 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485562 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-certs\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485852 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485625 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-socket\") pod \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\" (UID: \"ec32ea80-f52f-4588-8e2b-3d42aa4ee258\") " Nov 01 01:18:29.485852 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485718 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-data" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:29.485852 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485776 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istiod-ca-cert" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:29.486020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485957 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-certs" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:29.486020 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485998 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-socket" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:29.486146 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.486128 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.486207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.486153 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.486207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.486167 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.486207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.486181 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.486207 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.485926 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-credential-socket" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:29.487833 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.487793 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-token" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:29.487988 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.487920 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-envoy" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:29.488114 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.488092 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-podinfo" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:18:29.488241 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.488223 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-kube-api-access-hgdrb" pod "ec32ea80-f52f-4588-8e2b-3d42aa4ee258" (UID: "ec32ea80-f52f-4588-8e2b-3d42aa4ee258"). InnerVolumeSpecName "kube-api-access-hgdrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:29.587086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.587050 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.587086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.587086 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgdrb\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-kube-api-access-hgdrb\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.587383 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.587102 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.587383 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.587120 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.587383 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.587131 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec32ea80-f52f-4588-8e2b-3d42aa4ee258-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:29.797815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:29.797773 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:30.155642 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.155550 6036 generic.go:356] "Generic (PLEG): container finished" podID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerID="73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702" exitCode=0 Nov 01 01:18:30.155642 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.155601 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" Nov 01 01:18:30.155827 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.155643 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" event={"ID":"ec32ea80-f52f-4588-8e2b-3d42aa4ee258","Type":"ContainerDied","Data":"73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702"} Nov 01 01:18:30.155827 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.155702 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4" event={"ID":"ec32ea80-f52f-4588-8e2b-3d42aa4ee258","Type":"ContainerDied","Data":"225941ca4337bb5aed6b7728de893988218d6cc686fc5af71a97934753f93edd"} Nov 01 01:18:30.155827 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.155726 6036 scope.go:122] "RemoveContainer" containerID="73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702" Nov 01 01:18:30.165241 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.165226 6036 scope.go:122] "RemoveContainer" containerID="f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2" Nov 01 01:18:30.173977 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.173958 6036 scope.go:122] "RemoveContainer" containerID="73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702" Nov 01 01:18:30.174242 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:30.174219 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702\": container with ID starting with 73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702 not found: ID does not exist" containerID="73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702" Nov 01 01:18:30.174340 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.174254 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702"} err="failed to get container status \"73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702\": rpc error: code = NotFound desc = could not find container \"73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702\": container with ID starting with 73e915b297784e235ec462e5002632f595e03231651495e882517e8a8122d702 not found: ID does not exist" Nov 01 01:18:30.174340 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.174279 6036 scope.go:122] "RemoveContainer" containerID="f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2" Nov 01 01:18:30.174542 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:30.174524 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2\": container with ID starting with f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2 not found: ID does not exist" containerID="f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2" Nov 01 01:18:30.174584 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.174549 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2"} err="failed to get container status \"f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2\": rpc error: code = NotFound desc = could not find container \"f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2\": container with ID starting with f7ad9def88c7e43c6e2f3f2bb1e5102f061626fe3b055b639a730e4d97bb1ac2 not found: ID does not exist" Nov 01 01:18:30.374032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.373990 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4"] Nov 01 01:18:30.424441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:30.424361 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-48d289-openshift-default-64669d6567-f52n4"] Nov 01 01:18:31.249615 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:31.249576 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" path="/var/lib/kubelet/pods/ec32ea80-f52f-4588-8e2b-3d42aa4ee258/volumes" Nov 01 01:18:32.142731 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:32.142695 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:35.385723 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.385692 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:18:35.461724 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.461700 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:18:35.546101 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.546070 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15868285-24e8-4dfd-8700-9d20ec6f65c0-kube-api-access-sxsdr\" (UniqueName: \"kubernetes.io/projected/15868285-24e8-4dfd-8700-9d20ec6f65c0-kube-api-access-sxsdr\") pod \"15868285-24e8-4dfd-8700-9d20ec6f65c0\" (UID: \"15868285-24e8-4dfd-8700-9d20ec6f65c0\") " Nov 01 01:18:35.546328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.546174 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15868285-24e8-4dfd-8700-9d20ec6f65c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15868285-24e8-4dfd-8700-9d20ec6f65c0-plugins-config-volume\") pod \"15868285-24e8-4dfd-8700-9d20ec6f65c0\" (UID: \"15868285-24e8-4dfd-8700-9d20ec6f65c0\") " Nov 01 01:18:35.546542 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.546519 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15868285-24e8-4dfd-8700-9d20ec6f65c0-plugins-config-volume" pod "15868285-24e8-4dfd-8700-9d20ec6f65c0" (UID: "15868285-24e8-4dfd-8700-9d20ec6f65c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:35.548356 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.548336 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15868285-24e8-4dfd-8700-9d20ec6f65c0-kube-api-access-sxsdr" pod "15868285-24e8-4dfd-8700-9d20ec6f65c0" (UID: "15868285-24e8-4dfd-8700-9d20ec6f65c0"). InnerVolumeSpecName "kube-api-access-sxsdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:35.647581 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.647507 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-kube-api-access-whdsp\" (UniqueName: \"kubernetes.io/projected/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-kube-api-access-whdsp\") pod \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\" (UID: \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\") " Nov 01 01:18:35.647581 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.647576 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-plugins-config-volume\") pod \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\" (UID: \"9cb497c0-4e3b-46da-a860-c131fc8e7ec0\") " Nov 01 01:18:35.647853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.647841 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15868285-24e8-4dfd-8700-9d20ec6f65c0-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:35.647894 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.647854 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxsdr\" (UniqueName: \"kubernetes.io/projected/15868285-24e8-4dfd-8700-9d20ec6f65c0-kube-api-access-sxsdr\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:35.647984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.647952 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-plugins-config-volume" pod "9cb497c0-4e3b-46da-a860-c131fc8e7ec0" (UID: "9cb497c0-4e3b-46da-a860-c131fc8e7ec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:35.649808 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.649787 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-kube-api-access-whdsp" pod "9cb497c0-4e3b-46da-a860-c131fc8e7ec0" (UID: "9cb497c0-4e3b-46da-a860-c131fc8e7ec0"). InnerVolumeSpecName "kube-api-access-whdsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:35.748793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.748758 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whdsp\" (UniqueName: \"kubernetes.io/projected/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-kube-api-access-whdsp\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:35.748793 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:35.748787 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb497c0-4e3b-46da-a860-c131fc8e7ec0-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:36.186917 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.186876 6036 generic.go:356] "Generic (PLEG): container finished" podID="9cb497c0-4e3b-46da-a860-c131fc8e7ec0" containerID="d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201" exitCode=0 Nov 01 01:18:36.187113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.186928 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" Nov 01 01:18:36.187113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.186951 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" event={"ID":"9cb497c0-4e3b-46da-a860-c131fc8e7ec0","Type":"ContainerDied","Data":"d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201"} Nov 01 01:18:36.187113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.187012 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj" event={"ID":"9cb497c0-4e3b-46da-a860-c131fc8e7ec0","Type":"ContainerDied","Data":"cb410ef953193a152eda6e48c1ed53e2ac3c02e461887226cb23f02d0662af81"} Nov 01 01:18:36.187113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.187041 6036 scope.go:122] "RemoveContainer" containerID="d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201" Nov 01 01:18:36.188593 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.188568 6036 generic.go:356] "Generic (PLEG): container finished" podID="15868285-24e8-4dfd-8700-9d20ec6f65c0" containerID="b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d" exitCode=0 Nov 01 01:18:36.188688 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.188632 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" event={"ID":"15868285-24e8-4dfd-8700-9d20ec6f65c0","Type":"ContainerDied","Data":"b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d"} Nov 01 01:18:36.188688 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.188662 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" Nov 01 01:18:36.188688 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.188673 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw" event={"ID":"15868285-24e8-4dfd-8700-9d20ec6f65c0","Type":"ContainerDied","Data":"b31d3d95dd25809db486c6252c0e63e9916a3f2447b240346456a363aa45c239"} Nov 01 01:18:36.197930 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.197912 6036 scope.go:122] "RemoveContainer" containerID="d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201" Nov 01 01:18:36.198207 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:36.198184 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201\": container with ID starting with d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201 not found: ID does not exist" containerID="d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201" Nov 01 01:18:36.198316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.198214 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201"} err="failed to get container status \"d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201\": rpc error: code = NotFound desc = could not find container \"d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201\": container with ID starting with d453cf12245445a4e8a4322654af5014726242851bbe8d6df4c23c2f1de11201 not found: ID does not exist" Nov 01 01:18:36.198316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.198230 6036 scope.go:122] "RemoveContainer" containerID="b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d" Nov 01 01:18:36.206454 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.206434 6036 scope.go:122] "RemoveContainer" containerID="b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d" Nov 01 01:18:36.206706 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:36.206689 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d\": container with ID starting with b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d not found: ID does not exist" containerID="b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d" Nov 01 01:18:36.206784 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.206712 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d"} err="failed to get container status \"b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d\": rpc error: code = NotFound desc = could not find container \"b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d\": container with ID starting with b5eb0c420dce508eeb8bf895fb7e9bd7d86c3e117795460d1360a523344df53d not found: ID does not exist" Nov 01 01:18:36.359244 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.359209 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw"] Nov 01 01:18:36.409853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.409774 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d289-866ff47676-8qvvw"] Nov 01 01:18:36.484205 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.484174 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj"] Nov 01 01:18:36.490556 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:36.490527 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d289-dc7fb86f9-dlfnj"] Nov 01 01:18:37.098304 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:37.098257 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:37.251579 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:37.251543 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15868285-24e8-4dfd-8700-9d20ec6f65c0" path="/var/lib/kubelet/pods/15868285-24e8-4dfd-8700-9d20ec6f65c0/volumes" Nov 01 01:18:37.252066 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:37.252044 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb497c0-4e3b-46da-a860-c131fc8e7ec0" path="/var/lib/kubelet/pods/9cb497c0-4e3b-46da-a860-c131fc8e7ec0/volumes" Nov 01 01:18:37.309120 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:37.309086 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:39.208640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:39.208602 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:49.506789 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:49.506752 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:18:51.309419 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.309381 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk"] Nov 01 01:18:51.309840 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.309615 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" podUID="e957015c-79f2-4e43-b7dd-cc89e32169ba" containerName="maas-api" containerID="cri-o://60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b" gracePeriod=30 Nov 01 01:18:51.445939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.445900 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" podUID="e957015c-79f2-4e43-b7dd-cc89e32169ba" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Nov 01 01:18:51.478917 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.478838 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:51.558075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.558046 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 01:18:51.599696 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.599620 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e957015c-79f2-4e43-b7dd-cc89e32169ba-kube-api-access-q6knh\" (UniqueName: \"kubernetes.io/projected/e957015c-79f2-4e43-b7dd-cc89e32169ba-kube-api-access-q6knh\") pod \"e957015c-79f2-4e43-b7dd-cc89e32169ba\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " Nov 01 01:18:51.599885 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.599720 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e957015c-79f2-4e43-b7dd-cc89e32169ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e957015c-79f2-4e43-b7dd-cc89e32169ba-openshift-service-ca\") pod \"e957015c-79f2-4e43-b7dd-cc89e32169ba\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " Nov 01 01:18:51.599885 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.599775 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls\") pod \"e957015c-79f2-4e43-b7dd-cc89e32169ba\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " Nov 01 01:18:51.600001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.599891 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-metrics-certs\") pod \"e957015c-79f2-4e43-b7dd-cc89e32169ba\" (UID: \"e957015c-79f2-4e43-b7dd-cc89e32169ba\") " Nov 01 01:18:51.600135 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.600105 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e957015c-79f2-4e43-b7dd-cc89e32169ba-openshift-service-ca" pod "e957015c-79f2-4e43-b7dd-cc89e32169ba" (UID: "e957015c-79f2-4e43-b7dd-cc89e32169ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:51.600316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.600203 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e957015c-79f2-4e43-b7dd-cc89e32169ba-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:51.602041 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.602014 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls" pod "e957015c-79f2-4e43-b7dd-cc89e32169ba" (UID: "e957015c-79f2-4e43-b7dd-cc89e32169ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:18:51.602153 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.602039 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e957015c-79f2-4e43-b7dd-cc89e32169ba-kube-api-access-q6knh" pod "e957015c-79f2-4e43-b7dd-cc89e32169ba" (UID: "e957015c-79f2-4e43-b7dd-cc89e32169ba"). InnerVolumeSpecName "kube-api-access-q6knh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:51.602153 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.602108 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-metrics-certs" pod "e957015c-79f2-4e43-b7dd-cc89e32169ba" (UID: "e957015c-79f2-4e43-b7dd-cc89e32169ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:18:51.701369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.701336 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:51.701369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.701362 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e957015c-79f2-4e43-b7dd-cc89e32169ba-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:51.701369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:51.701374 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6knh\" (UniqueName: \"kubernetes.io/projected/e957015c-79f2-4e43-b7dd-cc89e32169ba-kube-api-access-q6knh\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:52.270408 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.270369 6036 generic.go:356] "Generic (PLEG): container finished" podID="e957015c-79f2-4e43-b7dd-cc89e32169ba" containerID="60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b" exitCode=0 Nov 01 01:18:52.270585 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.270416 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" Nov 01 01:18:52.270585 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.270451 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" event={"ID":"e957015c-79f2-4e43-b7dd-cc89e32169ba","Type":"ContainerDied","Data":"60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b"} Nov 01 01:18:52.270585 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.270526 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk" event={"ID":"e957015c-79f2-4e43-b7dd-cc89e32169ba","Type":"ContainerDied","Data":"025320ff6e61e1578fca14d8a25823259c46a647d739caa7a53360362b5cee93"} Nov 01 01:18:52.270585 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.270575 6036 scope.go:122] "RemoveContainer" containerID="60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b" Nov 01 01:18:52.281869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.281848 6036 scope.go:122] "RemoveContainer" containerID="60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b" Nov 01 01:18:52.282164 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:18:52.282143 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b\": container with ID starting with 60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b not found: ID does not exist" containerID="60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b" Nov 01 01:18:52.282229 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.282176 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b"} err="failed to get container status \"60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b\": rpc error: code = NotFound desc = could not find container \"60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b\": container with ID starting with 60d16b7b6295adea63847d56c941d845308946dd2fc89822cba6f8b2881c280b not found: ID does not exist" Nov 01 01:18:52.436859 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.436826 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk"] Nov 01 01:18:52.488345 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:52.488277 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-43a0cf-55455df9b4-5d9sk"] Nov 01 01:18:53.065165 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:53.065134 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:53.249557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:53.249524 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e957015c-79f2-4e43-b7dd-cc89e32169ba" path="/var/lib/kubelet/pods/e957015c-79f2-4e43-b7dd-cc89e32169ba/volumes" Nov 01 01:18:53.296201 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:53.296166 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:53.506253 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:53.506165 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:54.058189 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:54.058162 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:18:54.281323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:18:54.281259 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:00.874226 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:00.874191 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:01.173123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:01.173033 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4"] Nov 01 01:19:01.173342 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:01.173267 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" containerID="cri-o://d51a918e5c2a7c00fb7de93e1567ff916919182b54108078b63c08f843030ce3" gracePeriod=30 Nov 01 01:19:01.175987 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:01.175956 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:01.216427 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:01.216395 6036 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:19:01.216616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:01.216474 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:19:01.415469 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:01.415428 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:06.343919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.343892 6036 generic.go:356] "Generic (PLEG): container finished" podID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerID="d51a918e5c2a7c00fb7de93e1567ff916919182b54108078b63c08f843030ce3" exitCode=0 Nov 01 01:19:06.344461 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.343963 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" event={"ID":"75c9ed19-e38a-4bf4-859b-9dde6bf7901e","Type":"ContainerDied","Data":"d51a918e5c2a7c00fb7de93e1567ff916919182b54108078b63c08f843030ce3"} Nov 01 01:19:06.344461 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.344024 6036 scope.go:122] "RemoveContainer" containerID="4c92376e8ea155a66c9de70f76d169079d6f8d5d4605fa260f76a9d69789dcab" Nov 01 01:19:06.425848 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.425823 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 01:19:06.557544 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557503 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-credential-socket\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.557746 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557598 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-podinfo\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.557746 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557638 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-kube-api-access-lt8rg\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-kube-api-access-lt8rg\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.557746 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557671 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istiod-ca-cert\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.557746 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557731 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-envoy\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.557915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557774 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-token\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-token\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.557915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557825 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-socket\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.557915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557836 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-credential-socket" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:19:06.557915 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557893 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-certs\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.558136 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.557921 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-data\") pod \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\" (UID: \"75c9ed19-e38a-4bf4-859b-9dde6bf7901e\") " Nov 01 01:19:06.558136 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.558064 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istiod-ca-cert" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:19:06.558248 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.558190 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-certs" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:19:06.558387 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.558355 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-socket" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:19:06.558471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.558393 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:06.558471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.558416 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:06.558471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.558394 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-data" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:19:06.558654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.558432 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:06.560133 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.560103 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-podinfo" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:19:06.560133 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.560114 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-token" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:19:06.560354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.560182 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-kube-api-access-lt8rg" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "kube-api-access-lt8rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:19:06.560354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.560218 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-envoy" pod "75c9ed19-e38a-4bf4-859b-9dde6bf7901e" (UID: "75c9ed19-e38a-4bf4-859b-9dde6bf7901e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:19:06.659484 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.659449 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:06.659484 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.659475 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:06.659484 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.659486 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:06.659727 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.659494 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:06.659727 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.659504 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:06.659727 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:06.659513 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8rg\" (UniqueName: \"kubernetes.io/projected/75c9ed19-e38a-4bf4-859b-9dde6bf7901e-kube-api-access-lt8rg\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:07.349667 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:07.349633 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" event={"ID":"75c9ed19-e38a-4bf4-859b-9dde6bf7901e","Type":"ContainerDied","Data":"82662597c7205b761b4be59393c0a250282ad55a317e75b36cfd5d16a680b3c1"} Nov 01 01:19:07.349667 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:07.349657 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4" Nov 01 01:19:07.350167 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:07.349695 6036 scope.go:122] "RemoveContainer" containerID="d51a918e5c2a7c00fb7de93e1567ff916919182b54108078b63c08f843030ce3" Nov 01 01:19:07.566874 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:07.566840 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4"] Nov 01 01:19:07.617096 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:07.617060 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-6mmw4"] Nov 01 01:19:08.193630 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.193599 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-jdzvk"] Nov 01 01:19:08.194075 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194057 6036 cpu_manager.go:414] "containerMap: removing container" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194077 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194102 6036 cpu_manager.go:414] "containerMap: removing container" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194109 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194123 6036 cpu_manager.go:414] "containerMap: removing container" podUID="15868285-24e8-4dfd-8700-9d20ec6f65c0" containerName="payload-processing" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194131 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15868285-24e8-4dfd-8700-9d20ec6f65c0" containerName="payload-processing" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194142 6036 cpu_manager.go:414] "containerMap: removing container" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194150 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194163 6036 cpu_manager.go:414] "containerMap: removing container" podUID="e957015c-79f2-4e43-b7dd-cc89e32169ba" containerName="maas-api" Nov 01 01:19:08.194174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194171 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e957015c-79f2-4e43-b7dd-cc89e32169ba" containerName="maas-api" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194194 6036 cpu_manager.go:414] "containerMap: removing container" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194200 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194212 6036 cpu_manager.go:414] "containerMap: removing container" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194218 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194229 6036 cpu_manager.go:414] "containerMap: removing container" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194235 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194263 6036 cpu_manager.go:414] "containerMap: removing container" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194271 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194313 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9cb497c0-4e3b-46da-a860-c131fc8e7ec0" containerName="payload-pre-processing" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194322 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb497c0-4e3b-46da-a860-c131fc8e7ec0" containerName="payload-pre-processing" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194333 6036 cpu_manager.go:414] "containerMap: removing container" podUID="80eca316-5f74-41f7-9ac7-2ca9c73913ff" containerName="maas-api" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194340 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80eca316-5f74-41f7-9ac7-2ca9c73913ff" containerName="maas-api" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194350 6036 cpu_manager.go:414] "containerMap: removing container" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194357 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194379 6036 cpu_manager.go:414] "containerMap: removing container" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194386 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194490 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194501 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194513 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="da902477-5e58-4752-b788-4488ce5821c5" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194522 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb497c0-4e3b-46da-a860-c131fc8e7ec0" containerName="payload-pre-processing" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194533 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="80eca316-5f74-41f7-9ac7-2ca9c73913ff" containerName="maas-api" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194541 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194551 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="95829749-19bd-4ac0-86e8-a3635a805ea7" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194561 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf52ec0-ebbf-44ac-bc3e-dae9caac2d47" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194570 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde2721d-f699-4371-b0e8-584978a0df59" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194580 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="15868285-24e8-4dfd-8700-9d20ec6f65c0" containerName="payload-processing" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194591 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="caec49be-2da6-47ae-aad3-9ec0a797c6f8" containerName="istio-proxy" Nov 01 01:19:08.194678 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.194600 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e957015c-79f2-4e43-b7dd-cc89e32169ba" containerName="maas-api" Nov 01 01:19:08.199404 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.199382 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.252852 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.252826 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 01:19:08.303338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.303303 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 01:19:08.303338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.303304 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 01:19:08.303607 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.303305 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wgq79\"" Nov 01 01:19:08.305152 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.305121 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-jdzvk"] Nov 01 01:19:08.377410 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.377378 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.377840 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.377540 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99s4\" (UniqueName: \"kubernetes.io/projected/398e1661-4b76-4392-a84c-9f0e8373627b-kube-api-access-z99s4\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.377840 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.377568 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.478886 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.478794 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99s4\" (UniqueName: \"kubernetes.io/projected/398e1661-4b76-4392-a84c-9f0e8373627b-kube-api-access-z99s4\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.478886 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.478833 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.478886 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.478876 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.481524 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.481494 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.481676 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.481583 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.541346 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.541306 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99s4\" (UniqueName: \"kubernetes.io/projected/398e1661-4b76-4392-a84c-9f0e8373627b-kube-api-access-z99s4\") pod \"maas-controller-5fb6cd4f9b-jdzvk\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:08.812682 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:08.812646 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:09.187205 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:09.187141 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-jdzvk"] Nov 01 01:19:09.189532 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:19:09.189493 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398e1661_4b76_4392_a84c_9f0e8373627b.slice/crio-571b6c42ad55016017cad5b07e4816363e60946bdba02ea37be7b039460dcba1 WatchSource:0}: Error finding container 571b6c42ad55016017cad5b07e4816363e60946bdba02ea37be7b039460dcba1: Status 404 returned error can't find the container with id 571b6c42ad55016017cad5b07e4816363e60946bdba02ea37be7b039460dcba1 Nov 01 01:19:09.190732 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:09.190715 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 01:19:09.249795 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:09.249761 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c9ed19-e38a-4bf4-859b-9dde6bf7901e" path="/var/lib/kubelet/pods/75c9ed19-e38a-4bf4-859b-9dde6bf7901e/volumes" Nov 01 01:19:09.361263 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:09.361222 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" event={"ID":"398e1661-4b76-4392-a84c-9f0e8373627b","Type":"ContainerStarted","Data":"571b6c42ad55016017cad5b07e4816363e60946bdba02ea37be7b039460dcba1"} Nov 01 01:19:11.373508 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:11.373473 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" event={"ID":"398e1661-4b76-4392-a84c-9f0e8373627b","Type":"ContainerStarted","Data":"4b37de784988f42e01bc56accfcdb912b75b255497e49afb77bacd6b19f2ef65"} Nov 01 01:19:11.373916 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:11.373560 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:11.494946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:11.494844 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" podStartSLOduration=1.484562964 podStartE2EDuration="3.49482909s" totalImagesPullingTime="2.010266126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:19:11.493583809 +0000 UTC m=+3296.368804824" watchObservedRunningTime="2026-11-01 01:19:11.49482909 +0000 UTC m=+3296.370050104" Nov 01 01:19:16.221568 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:16.221529 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:18.566693 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:18.566655 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:19.503590 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:19:19.503554 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:19:22.381383 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:22.381343 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:38.805302 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.805251 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f"] Nov 01 01:19:38.805885 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.805867 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec32ea80-f52f-4588-8e2b-3d42aa4ee258" containerName="istio-proxy" Nov 01 01:19:38.810750 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.810724 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.863995 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.863963 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srhv5\"" Nov 01 01:19:38.863995 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.863992 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 01:19:38.874954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.874927 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.875068 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.874987 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a82f5c-24ab-4385-9b59-832fec6461a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.875109 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.875082 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.875154 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.875141 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.875521 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.875267 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.875521 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.875344 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgdr6\" (UniqueName: \"kubernetes.io/projected/12a82f5c-24ab-4385-9b59-832fec6461a2-kube-api-access-lgdr6\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.875521 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.875378 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.875521 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.875476 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.875722 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.875524 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.916062 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.916031 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f"] Nov 01 01:19:38.976461 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976422 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.976645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976473 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.976645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976518 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.976645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976561 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a82f5c-24ab-4385-9b59-832fec6461a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.976645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976594 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.976866 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976656 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.976866 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976786 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.976866 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976842 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdr6\" (UniqueName: \"kubernetes.io/projected/12a82f5c-24ab-4385-9b59-832fec6461a2-kube-api-access-lgdr6\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.977024 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976880 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.977024 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.976962 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.977169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.977134 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.977263 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.977223 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.977263 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.977246 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.977394 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.977375 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a82f5c-24ab-4385-9b59-832fec6461a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.979179 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.979155 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:38.979420 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:38.979402 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:39.038821 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:39.038791 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a82f5c-24ab-4385-9b59-832fec6461a2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:39.039016 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:39.038902 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdr6\" (UniqueName: \"kubernetes.io/projected/12a82f5c-24ab-4385-9b59-832fec6461a2-kube-api-access-lgdr6\") pod \"maas-default-gateway-openshift-default-67bf986f99-w4d6f\" (UID: \"12a82f5c-24ab-4385-9b59-832fec6461a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:39.124118 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:39.124030 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:39.552919 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:39.552745 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f"] Nov 01 01:19:39.555251 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:19:39.555223 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a82f5c_24ab_4385_9b59_832fec6461a2.slice/crio-4a5731b71e216e93689ab860b39aa57981b31d6aff39cb39224e469fc44ef8e4 WatchSource:0}: Error finding container 4a5731b71e216e93689ab860b39aa57981b31d6aff39cb39224e469fc44ef8e4: Status 404 returned error can't find the container with id 4a5731b71e216e93689ab860b39aa57981b31d6aff39cb39224e469fc44ef8e4 Nov 01 01:19:39.557566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:39.557522 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:19:39.557688 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:39.557616 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:19:39.557688 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:39.557652 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:19:39.628668 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:39.628634 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:40.513823 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:40.513784 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" event={"ID":"12a82f5c-24ab-4385-9b59-832fec6461a2","Type":"ContainerStarted","Data":"f0736ad73e98ef1fc9b939d846e89179cf059faf4dbe7a157cfa3147d7fd2d2e"} Nov 01 01:19:40.514311 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:40.513840 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" event={"ID":"12a82f5c-24ab-4385-9b59-832fec6461a2","Type":"ContainerStarted","Data":"4a5731b71e216e93689ab860b39aa57981b31d6aff39cb39224e469fc44ef8e4"} Nov 01 01:19:40.687304 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:40.687245 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" podStartSLOduration=2.687229711 podStartE2EDuration="2.687229711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:19:40.68621032 +0000 UTC m=+3325.561431327" watchObservedRunningTime="2026-11-01 01:19:40.687229711 +0000 UTC m=+3325.562450726" Nov 01 01:19:41.124837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:41.124793 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:41.126318 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:41.126273 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-w4d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 01:19:41.126380 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:41.126344 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" podUID="12a82f5c-24ab-4385-9b59-832fec6461a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 01:19:42.124855 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:42.124815 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-w4d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 01:19:42.125278 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:42.124868 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" podUID="12a82f5c-24ab-4385-9b59-832fec6461a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 01:19:43.124413 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:43.124378 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-w4d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 01:19:43.124575 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:43.124429 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" podUID="12a82f5c-24ab-4385-9b59-832fec6461a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 01:19:43.893177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:43.893139 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:43.962638 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:43.962582 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-jdzvk"] Nov 01 01:19:43.963658 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:43.963623 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" podUID="398e1661-4b76-4392-a84c-9f0e8373627b" containerName="manager" containerID="cri-o://4b37de784988f42e01bc56accfcdb912b75b255497e49afb77bacd6b19f2ef65" gracePeriod=30 Nov 01 01:19:44.125303 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.125261 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-w4d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 01:19:44.125489 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.125331 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" podUID="12a82f5c-24ab-4385-9b59-832fec6461a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 01:19:44.534714 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.534678 6036 generic.go:356] "Generic (PLEG): container finished" podID="398e1661-4b76-4392-a84c-9f0e8373627b" containerID="4b37de784988f42e01bc56accfcdb912b75b255497e49afb77bacd6b19f2ef65" exitCode=0 Nov 01 01:19:44.534911 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.534757 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" event={"ID":"398e1661-4b76-4392-a84c-9f0e8373627b","Type":"ContainerDied","Data":"4b37de784988f42e01bc56accfcdb912b75b255497e49afb77bacd6b19f2ef65"} Nov 01 01:19:44.737626 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.737595 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:44.832176 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.832089 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398e1661-4b76-4392-a84c-9f0e8373627b-kube-api-access-z99s4\" (UniqueName: \"kubernetes.io/projected/398e1661-4b76-4392-a84c-9f0e8373627b-kube-api-access-z99s4\") pod \"398e1661-4b76-4392-a84c-9f0e8373627b\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " Nov 01 01:19:44.832176 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.832167 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-webhook-cert\") pod \"398e1661-4b76-4392-a84c-9f0e8373627b\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " Nov 01 01:19:44.832493 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.832277 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-metrics-certs\") pod \"398e1661-4b76-4392-a84c-9f0e8373627b\" (UID: \"398e1661-4b76-4392-a84c-9f0e8373627b\") " Nov 01 01:19:44.834555 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.834525 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398e1661-4b76-4392-a84c-9f0e8373627b-kube-api-access-z99s4" pod "398e1661-4b76-4392-a84c-9f0e8373627b" (UID: "398e1661-4b76-4392-a84c-9f0e8373627b"). InnerVolumeSpecName "kube-api-access-z99s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:19:44.834655 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.834560 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-webhook-cert" pod "398e1661-4b76-4392-a84c-9f0e8373627b" (UID: "398e1661-4b76-4392-a84c-9f0e8373627b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:19:44.834655 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.834600 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-metrics-certs" pod "398e1661-4b76-4392-a84c-9f0e8373627b" (UID: "398e1661-4b76-4392-a84c-9f0e8373627b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:19:44.933417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.933384 6036 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-webhook-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:44.933417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.933413 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e1661-4b76-4392-a84c-9f0e8373627b-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:44.933417 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:44.933424 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99s4\" (UniqueName: \"kubernetes.io/projected/398e1661-4b76-4392-a84c-9f0e8373627b-kube-api-access-z99s4\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:45.125392 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:45.125279 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-w4d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 01:19:45.125392 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:45.125365 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" podUID="12a82f5c-24ab-4385-9b59-832fec6461a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 01:19:45.541441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:45.541405 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" event={"ID":"398e1661-4b76-4392-a84c-9f0e8373627b","Type":"ContainerDied","Data":"571b6c42ad55016017cad5b07e4816363e60946bdba02ea37be7b039460dcba1"} Nov 01 01:19:45.541441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:45.541422 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-jdzvk" Nov 01 01:19:45.541705 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:45.541467 6036 scope.go:122] "RemoveContainer" containerID="4b37de784988f42e01bc56accfcdb912b75b255497e49afb77bacd6b19f2ef65" Nov 01 01:19:45.708626 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:45.708588 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-jdzvk"] Nov 01 01:19:45.713674 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:45.713650 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-jdzvk"] Nov 01 01:19:46.124763 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:46.124727 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-w4d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 01:19:46.125183 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:46.124781 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" podUID="12a82f5c-24ab-4385-9b59-832fec6461a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 01:19:47.127703 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.127668 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:47.200517 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.200477 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-frmhp"] Nov 01 01:19:47.201128 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.201107 6036 cpu_manager.go:414] "containerMap: removing container" podUID="398e1661-4b76-4392-a84c-9f0e8373627b" containerName="manager" Nov 01 01:19:47.201173 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.201134 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398e1661-4b76-4392-a84c-9f0e8373627b" containerName="manager" Nov 01 01:19:47.201347 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.201333 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="398e1661-4b76-4392-a84c-9f0e8373627b" containerName="manager" Nov 01 01:19:47.206245 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.206219 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.249594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.249555 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398e1661-4b76-4392-a84c-9f0e8373627b" path="/var/lib/kubelet/pods/398e1661-4b76-4392-a84c-9f0e8373627b/volumes" Nov 01 01:19:47.310188 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.310145 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 01:19:47.310188 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.310148 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 01:19:47.310469 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.310275 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 01:19:47.312476 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.312450 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wgq79\"" Nov 01 01:19:47.314186 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.314157 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-frmhp"] Nov 01 01:19:47.359174 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.359134 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cxvf\" (UniqueName: \"kubernetes.io/projected/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-kube-api-access-5cxvf\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.359405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.359186 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.359537 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.359506 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.460839 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.460730 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.461046 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.460870 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.461046 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.460898 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxvf\" (UniqueName: \"kubernetes.io/projected/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-kube-api-access-5cxvf\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.463583 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.463553 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.463804 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.463778 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.529242 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.529201 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cxvf\" (UniqueName: \"kubernetes.io/projected/7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7-kube-api-access-5cxvf\") pod \"maas-controller-5fb6cd4f9b-frmhp\" (UID: \"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7\") " pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:47.796453 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.796417 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:47.796891 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.796776 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:47.797685 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.797665 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-w4d6f" Nov 01 01:19:47.819441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:47.819414 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:48.194224 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:48.194196 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-frmhp"] Nov 01 01:19:48.196927 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:19:48.196902 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0e68c4_64bc_4c2b_b35f_1768b2fbf9e7.slice/crio-cee745ffb699ddadc26a63e940df5a3e51e164a23e617e60b7957077cb611127 WatchSource:0}: Error finding container cee745ffb699ddadc26a63e940df5a3e51e164a23e617e60b7957077cb611127: Status 404 returned error can't find the container with id cee745ffb699ddadc26a63e940df5a3e51e164a23e617e60b7957077cb611127 Nov 01 01:19:48.559633 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:48.559589 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" event={"ID":"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7","Type":"ContainerStarted","Data":"cee745ffb699ddadc26a63e940df5a3e51e164a23e617e60b7957077cb611127"} Nov 01 01:19:49.504534 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:19:49.504492 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:19:49.566134 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:49.566097 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" event={"ID":"7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7","Type":"ContainerStarted","Data":"faaba009d17670622a137faae6e24dae7e9066a4f12f9760fbc9b13d76dcf69a"} Nov 01 01:19:49.566349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:49.566333 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:19:49.687473 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:49.687426 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" podStartSLOduration=1.854236854 podStartE2EDuration="2.687412041s" totalImagesPullingTime="833.175187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:19:49.685483035 +0000 UTC m=+3334.560704066" watchObservedRunningTime="2026-11-01 01:19:49.687412041 +0000 UTC m=+3334.562633049" Nov 01 01:19:55.283371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:55.283252 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:55.490401 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:55.490366 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:19:55.759565 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:19:55.759463 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:20:00.574280 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:00.574246 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb6cd4f9b-frmhp" Nov 01 01:20:19.503126 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:20:19.503083 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:20:39.918306 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:39.918245 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:20:49.504434 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:20:49.504393 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:20:54.406355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.406321 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn"] Nov 01 01:20:54.410123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.410098 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.456819 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.456773 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.456819 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.456817 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqz8\" (UniqueName: \"kubernetes.io/projected/cb6d7ab3-42e6-40f5-8c71-1d375595043e-kube-api-access-hpqz8\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.457095 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.456923 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.457095 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.456978 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.457095 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.457025 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.457095 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.457054 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.457235 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.457101 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.457235 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.457118 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.457330 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.457242 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558247 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558542 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558358 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558542 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558424 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558542 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558486 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558542 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558514 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqz8\" (UniqueName: \"kubernetes.io/projected/cb6d7ab3-42e6-40f5-8c71-1d375595043e-kube-api-access-hpqz8\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558555 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558596 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558643 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.558779 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558676 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.559010 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558826 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.559010 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558930 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.559010 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.558979 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.559184 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.559103 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.559481 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.559460 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.561065 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.561044 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.561183 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.561166 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.614443 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.614362 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn"] Nov 01 01:20:54.620635 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.620604 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb6d7ab3-42e6-40f5-8c71-1d375595043e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.627306 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.621301 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqz8\" (UniqueName: \"kubernetes.io/projected/cb6d7ab3-42e6-40f5-8c71-1d375595043e-kube-api-access-hpqz8\") pod \"maas-default-gateway-openshift-default-67bf986f99-8jzfn\" (UID: \"cb6d7ab3-42e6-40f5-8c71-1d375595043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:54.722594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:54.722513 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:55.154653 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:55.154616 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn"] Nov 01 01:20:55.156150 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:20:55.156124 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6d7ab3_42e6_40f5_8c71_1d375595043e.slice/crio-278ccfe9e84ce11029a1ec70bb22b3e5b823e5efd6ec1ff479f7309cec0f3e1f WatchSource:0}: Error finding container 278ccfe9e84ce11029a1ec70bb22b3e5b823e5efd6ec1ff479f7309cec0f3e1f: Status 404 returned error can't find the container with id 278ccfe9e84ce11029a1ec70bb22b3e5b823e5efd6ec1ff479f7309cec0f3e1f Nov 01 01:20:55.158422 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:55.158388 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:20:55.158489 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:55.158479 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:20:55.158530 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:55.158510 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:20:55.897985 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:55.897951 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" event={"ID":"cb6d7ab3-42e6-40f5-8c71-1d375595043e","Type":"ContainerStarted","Data":"cdfe842d152c58ee161c6b84e13ccbf4f3110ce23d97f9599e2479525a078704"} Nov 01 01:20:55.898567 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:55.898013 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" event={"ID":"cb6d7ab3-42e6-40f5-8c71-1d375595043e","Type":"ContainerStarted","Data":"278ccfe9e84ce11029a1ec70bb22b3e5b823e5efd6ec1ff479f7309cec0f3e1f"} Nov 01 01:20:56.074687 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:56.074614 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podStartSLOduration=2.07459089 podStartE2EDuration="2.07459089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:20:56.072334264 +0000 UTC m=+3400.947555279" watchObservedRunningTime="2026-11-01 01:20:56.07459089 +0000 UTC m=+3400.949811905" Nov 01 01:20:56.722831 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:56.722797 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:20:56.724471 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:56.724444 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 01:20:56.724627 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:56.724499 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 01:20:57.723722 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:57.723685 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 01:20:57.724141 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:57.723740 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 01:20:58.723579 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:58.723541 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 01:20:58.723747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:58.723611 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 01:20:59.723861 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:59.723820 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 01:20:59.724262 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:20:59.723873 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 01:21:00.723908 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:00.723864 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 01:21:00.724466 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:00.723930 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 01:21:01.723273 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:01.723235 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 01:21:01.723521 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:01.723305 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 01:21:02.723617 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:02.723567 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 01:21:02.724184 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:02.723628 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 01:21:04.764478 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:04.764427 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 01:21:04.765012 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:04.764490 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 01:21:05.519771 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:05.519737 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:21:05.520228 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:05.520209 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:21:05.520929 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:05.520912 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" Nov 01 01:21:09.523228 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.523187 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb"] Nov 01 01:21:09.527439 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.527415 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626055 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626021 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626232 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626060 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626232 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626083 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626232 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626218 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626397 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626277 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e53743-67de-4052-be2e-997144f7a8c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626397 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626388 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e53743-67de-4052-be2e-997144f7a8c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626473 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626427 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626520 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626484 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-688v4\" (UniqueName: \"kubernetes.io/projected/d2e53743-67de-4052-be2e-997144f7a8c1-kube-api-access-688v4\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.626561 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.626531 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e53743-67de-4052-be2e-997144f7a8c1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728143 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728096 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e53743-67de-4052-be2e-997144f7a8c1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728156 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728178 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728205 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728243 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728330 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e53743-67de-4052-be2e-997144f7a8c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728420 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e53743-67de-4052-be2e-997144f7a8c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728482 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728517 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-688v4\" (UniqueName: \"kubernetes.io/projected/d2e53743-67de-4052-be2e-997144f7a8c1-kube-api-access-688v4\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728650 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.728738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728729 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.729042 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.728919 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.729126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.729101 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.729524 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.729449 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e53743-67de-4052-be2e-997144f7a8c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.730559 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.730527 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb"] Nov 01 01:21:09.731532 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.731504 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e53743-67de-4052-be2e-997144f7a8c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.731639 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.731527 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e53743-67de-4052-be2e-997144f7a8c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.790527 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.790493 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e53743-67de-4052-be2e-997144f7a8c1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.790527 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.790509 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-688v4\" (UniqueName: \"kubernetes.io/projected/d2e53743-67de-4052-be2e-997144f7a8c1-kube-api-access-688v4\") pod \"maas-default-gateway-openshift-default-67bf986f99-q4kwb\" (UID: \"d2e53743-67de-4052-be2e-997144f7a8c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:09.840862 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:09.840829 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:10.271543 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:10.271514 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb"] Nov 01 01:21:10.273753 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:21:10.273721 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e53743_67de_4052_be2e_997144f7a8c1.slice/crio-e27e17ef5a716dec9af7529312c7c268124beb84203575f4f997576a00691439 WatchSource:0}: Error finding container e27e17ef5a716dec9af7529312c7c268124beb84203575f4f997576a00691439: Status 404 returned error can't find the container with id e27e17ef5a716dec9af7529312c7c268124beb84203575f4f997576a00691439 Nov 01 01:21:10.275974 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:10.275944 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:21:10.276058 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:10.276002 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:21:10.276058 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:10.276029 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:21:10.976706 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:10.976672 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" event={"ID":"d2e53743-67de-4052-be2e-997144f7a8c1","Type":"ContainerStarted","Data":"9983616b2edc13ce5bc43c55307daceffaf18cbaac8d902166597e3a408cd9d6"} Nov 01 01:21:10.977241 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:10.976727 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" event={"ID":"d2e53743-67de-4052-be2e-997144f7a8c1","Type":"ContainerStarted","Data":"e27e17ef5a716dec9af7529312c7c268124beb84203575f4f997576a00691439"} Nov 01 01:21:11.841078 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:11.841047 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:11.842760 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:11.842734 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q4kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 01:21:11.842923 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:11.842798 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" podUID="d2e53743-67de-4052-be2e-997144f7a8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 01:21:12.841822 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:12.841783 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q4kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 01:21:12.842257 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:12.841844 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" podUID="d2e53743-67de-4052-be2e-997144f7a8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 01:21:13.842249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:13.842212 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q4kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 01:21:13.842709 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:13.842274 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" podUID="d2e53743-67de-4052-be2e-997144f7a8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 01:21:14.841869 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:14.841833 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q4kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 01:21:14.842071 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:14.841888 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" podUID="d2e53743-67de-4052-be2e-997144f7a8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 01:21:15.842153 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:15.842121 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q4kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 01:21:15.842614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:15.842172 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" podUID="d2e53743-67de-4052-be2e-997144f7a8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 01:21:16.841371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:16.841334 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q4kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 01:21:16.841558 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:16.841390 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" podUID="d2e53743-67de-4052-be2e-997144f7a8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 01:21:18.571917 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:18.571882 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:18.572480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:18.572175 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:18.573134 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:18.573112 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" Nov 01 01:21:18.745898 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:18.745847 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q4kwb" podStartSLOduration=9.745833799 podStartE2EDuration="9.745833799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:21:11.170470652 +0000 UTC m=+3416.045691668" watchObservedRunningTime="2026-11-01 01:21:18.745833799 +0000 UTC m=+3423.621054814" Nov 01 01:21:19.505013 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:21:19.504973 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:21:40.278600 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:40.278520 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:21:40.571803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:21:40.571768 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:21:49.504935 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:21:49.504886 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:22:19.505956 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:22:19.505912 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:22:40.638203 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:22:40.638168 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:22:49.504385 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:22:49.504347 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:23:19.504688 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:23:19.504649 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:23:40.967474 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:23:40.967433 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:23:49.504566 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:23:49.504522 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:24:07.707357 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:24:07.707317 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:24:07.918952 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:24:07.918916 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:24:16.110211 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:24:16.110166 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:24:19.504183 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:24:19.504144 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:24:27.231860 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:24:27.231772 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:24:38.462122 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:24:38.462081 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:24:41.393603 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:24:41.393570 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:24:49.505094 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:24:49.505028 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:25:19.505309 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:25:19.505251 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:25:41.674654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:25:41.674619 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:25:49.504309 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:25:49.504244 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:26:06.832025 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:06.831994 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:07.443410 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:07.443371 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:07.763745 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:07.763659 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:10.353148 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:10.353115 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:12.019279 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:12.019241 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:12.237591 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:12.237559 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:15.315587 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.315543 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:15.521388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.521357 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:15.820571 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.820531 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf"] Nov 01 01:26:15.824530 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.824505 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:15.878469 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.878436 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-43c48b\"" Nov 01 01:26:15.928802 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.928766 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-1ebacc3d-dockercfg-hxb6j\"" Nov 01 01:26:15.930308 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.930266 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf"] Nov 01 01:26:15.992193 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.992160 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7"] Nov 01 01:26:15.995699 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:15.995679 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:16.000084 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.000059 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cf5q\" (UniqueName: \"kubernetes.io/projected/5343d36c-a98b-4bfe-a118-c318d6c7d707-kube-api-access-9cf5q\") pod \"payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf\" (UID: \"5343d36c-a98b-4bfe-a118-c318d6c7d707\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:16.000299 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.000260 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5343d36c-a98b-4bfe-a118-c318d6c7d707-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf\" (UID: \"5343d36c-a98b-4bfe-a118-c318d6c7d707\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:16.054280 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.054237 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7"] Nov 01 01:26:16.101687 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.101597 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7\" (UID: \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:16.101687 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.101648 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5343d36c-a98b-4bfe-a118-c318d6c7d707-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf\" (UID: \"5343d36c-a98b-4bfe-a118-c318d6c7d707\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:16.101940 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.101799 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cf5q\" (UniqueName: \"kubernetes.io/projected/5343d36c-a98b-4bfe-a118-c318d6c7d707-kube-api-access-9cf5q\") pod \"payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf\" (UID: \"5343d36c-a98b-4bfe-a118-c318d6c7d707\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:16.101940 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.101856 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782ss\" (UniqueName: \"kubernetes.io/projected/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-kube-api-access-782ss\") pod \"payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7\" (UID: \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:16.102369 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.102330 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5343d36c-a98b-4bfe-a118-c318d6c7d707-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf\" (UID: \"5343d36c-a98b-4bfe-a118-c318d6c7d707\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:16.163924 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.163889 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cf5q\" (UniqueName: \"kubernetes.io/projected/5343d36c-a98b-4bfe-a118-c318d6c7d707-kube-api-access-9cf5q\") pod \"payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf\" (UID: \"5343d36c-a98b-4bfe-a118-c318d6c7d707\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:16.203446 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.203408 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7\" (UID: \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:16.203634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.203520 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782ss\" (UniqueName: \"kubernetes.io/projected/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-kube-api-access-782ss\") pod \"payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7\" (UID: \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:16.204077 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.204057 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7\" (UID: \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:16.266161 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.266126 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-782ss\" (UniqueName: \"kubernetes.io/projected/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-kube-api-access-782ss\") pod \"payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7\" (UID: \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:16.309125 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.309090 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:16.439046 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.439010 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:16.683244 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.683216 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7"] Nov 01 01:26:16.685510 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:26:16.685481 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb12ab6_f69a_46c2_86e9_cef7cb829fb4.slice/crio-57674c0a84bec6f1fc2185dac156f02bc0c467912d5cb05baead2db064f68d00 WatchSource:0}: Error finding container 57674c0a84bec6f1fc2185dac156f02bc0c467912d5cb05baead2db064f68d00: Status 404 returned error can't find the container with id 57674c0a84bec6f1fc2185dac156f02bc0c467912d5cb05baead2db064f68d00 Nov 01 01:26:16.814441 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:16.814415 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf"] Nov 01 01:26:16.816444 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:26:16.816413 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5343d36c_a98b_4bfe_a118_c318d6c7d707.slice/crio-58535f6c1e4368a7cd5639791377e2d66aeefc3ad6ee1735275f3ffb9831836a WatchSource:0}: Error finding container 58535f6c1e4368a7cd5639791377e2d66aeefc3ad6ee1735275f3ffb9831836a: Status 404 returned error can't find the container with id 58535f6c1e4368a7cd5639791377e2d66aeefc3ad6ee1735275f3ffb9831836a Nov 01 01:26:17.608187 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:17.608151 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" event={"ID":"5343d36c-a98b-4bfe-a118-c318d6c7d707","Type":"ContainerStarted","Data":"5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93"} Nov 01 01:26:17.608710 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:17.608205 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" event={"ID":"5343d36c-a98b-4bfe-a118-c318d6c7d707","Type":"ContainerStarted","Data":"58535f6c1e4368a7cd5639791377e2d66aeefc3ad6ee1735275f3ffb9831836a"} Nov 01 01:26:17.608710 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:17.608250 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:17.609948 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:17.609921 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" event={"ID":"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4","Type":"ContainerStarted","Data":"251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3"} Nov 01 01:26:17.610058 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:17.609967 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" event={"ID":"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4","Type":"ContainerStarted","Data":"57674c0a84bec6f1fc2185dac156f02bc0c467912d5cb05baead2db064f68d00"} Nov 01 01:26:17.610058 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:17.609989 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:17.728331 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:17.727146 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" podStartSLOduration=2.727126037 podStartE2EDuration="2.727126037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:26:17.725049202 +0000 UTC m=+3722.600270231" watchObservedRunningTime="2026-11-01 01:26:17.727126037 +0000 UTC m=+3722.602347052" Nov 01 01:26:17.842642 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:17.842584 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" podStartSLOduration=2.8425709919999997 podStartE2EDuration="2.842570992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:26:17.842324464 +0000 UTC m=+3722.717545483" watchObservedRunningTime="2026-11-01 01:26:17.842570992 +0000 UTC m=+3722.717792007" Nov 01 01:26:18.018267 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.018187 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr"] Nov 01 01:26:18.021894 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.021874 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.075883 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.075849 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-43c48b-serving-cert\"" Nov 01 01:26:18.081990 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.081917 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr"] Nov 01 01:26:18.122439 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.122399 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjfsg\" (UniqueName: \"kubernetes.io/projected/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-kube-api-access-mjfsg\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.122617 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.122443 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.122617 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.122471 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.122711 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.122618 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.223550 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.223510 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.223737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.223613 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjfsg\" (UniqueName: \"kubernetes.io/projected/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-kube-api-access-mjfsg\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.223737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.223637 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.223737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.223652 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.223737 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:26:18.223662 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-serving-cert: secret "maas-api-e2e-worker-single-main-43c48b-serving-cert" not found Nov 01 01:26:18.223912 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:26:18.223754 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls podName:29baf82c-6242-44a4-a27d-e4c7a4bd24c8 nodeName:}" failed. No retries permitted until 2026-11-01 01:26:18.723730481 +0000 UTC m=+3723.598951487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls") pod "maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" (UID: "29baf82c-6242-44a4-a27d-e4c7a4bd24c8") : secret "maas-api-e2e-worker-single-main-43c48b-serving-cert" not found Nov 01 01:26:18.224313 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.224277 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.226394 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.226369 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.282503 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.282477 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:18.286031 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.286000 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjfsg\" (UniqueName: \"kubernetes.io/projected/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-kube-api-access-mjfsg\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.489759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.489720 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:18.701580 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.701502 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:18.730225 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.730189 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.732889 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.732863 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:18.935899 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:18.935862 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:19.310628 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:19.310603 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr"] Nov 01 01:26:19.312771 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:26:19.312741 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29baf82c_6242_44a4_a27d_e4c7a4bd24c8.slice/crio-f23466b7e38ad4a9f471668244feed2ff81e97f32d1ff8aacb1255223aa5bc33 WatchSource:0}: Error finding container f23466b7e38ad4a9f471668244feed2ff81e97f32d1ff8aacb1255223aa5bc33: Status 404 returned error can't find the container with id f23466b7e38ad4a9f471668244feed2ff81e97f32d1ff8aacb1255223aa5bc33 Nov 01 01:26:19.314254 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:19.314236 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 01:26:19.505927 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:26:19.505891 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:26:19.621486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:19.621399 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" event={"ID":"29baf82c-6242-44a4-a27d-e4c7a4bd24c8","Type":"ContainerStarted","Data":"f23466b7e38ad4a9f471668244feed2ff81e97f32d1ff8aacb1255223aa5bc33"} Nov 01 01:26:20.627645 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:20.627609 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" event={"ID":"29baf82c-6242-44a4-a27d-e4c7a4bd24c8","Type":"ContainerStarted","Data":"b35a165ceceb6f1ab05ab095522ef97b49ee5578d92c963b68975ff18abd48cb"} Nov 01 01:26:20.628045 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:20.627675 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:20.746516 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:20.746466 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" podStartSLOduration=2.8130601669999997 podStartE2EDuration="3.74645141s" totalImagesPullingTime="933.391243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:26:20.746138909 +0000 UTC m=+3725.621359925" watchObservedRunningTime="2026-11-01 01:26:20.74645141 +0000 UTC m=+3725.621672424" Nov 01 01:26:23.231622 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:23.231588 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:26.638478 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:26.638450 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:26:28.238497 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:28.238465 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:28.617999 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:28.617973 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:26:28.618164 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:28.618036 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:26:33.031308 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:33.031238 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:36.914036 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:36.913997 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:37.520721 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:37.520685 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:37.754882 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:37.754851 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:38.052835 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:38.052719 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:41.386901 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:41.386843 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:42.011546 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:42.011510 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:42.838064 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:42.838013 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:47.667151 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:47.667106 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:49.504429 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:26:49.504391 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:26:52.430346 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:52.430313 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:26:57.325633 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:26:57.325600 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:02.134132 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:02.134097 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:07.026563 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:07.026521 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:11.841629 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:11.841592 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:16.728235 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:16.728192 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:19.504188 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:27:19.504150 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:27:21.580012 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:21.579981 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:26.362183 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:26.362091 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:31.143263 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:31.143219 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:35.945405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:35.945362 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:36.759621 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:36.759570 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:41.144585 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:41.144538 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:41.791998 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:41.791966 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:42.359609 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:42.359575 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:46.141684 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:46.141645 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:48.750367 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:48.750332 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf"] Nov 01 01:27:48.857458 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:48.857425 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7"] Nov 01 01:27:49.273438 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:49.273391 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:49.492566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:49.492532 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:49.505396 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:27:49.505366 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:27:50.936279 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:50.936239 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:53.751316 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:53.751244 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" podUID="5343d36c-a98b-4bfe-a118-c318d6c7d707" containerName="payload-processing" containerID="cri-o://5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93" gracePeriod=25 Nov 01 01:27:53.858414 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:53.858353 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" podUID="0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" containerName="payload-pre-processing" containerID="cri-o://251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3" gracePeriod=25 Nov 01 01:27:54.995545 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:54.995511 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr"] Nov 01 01:27:54.995982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:54.995737 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" podUID="29baf82c-6242-44a4-a27d-e4c7a4bd24c8" containerName="maas-api" containerID="cri-o://b35a165ceceb6f1ab05ab095522ef97b49ee5578d92c963b68975ff18abd48cb" gracePeriod=30 Nov 01 01:27:55.160223 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:55.160181 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:56.111745 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.111714 6036 generic.go:356] "Generic (PLEG): container finished" podID="29baf82c-6242-44a4-a27d-e4c7a4bd24c8" containerID="b35a165ceceb6f1ab05ab095522ef97b49ee5578d92c963b68975ff18abd48cb" exitCode=0 Nov 01 01:27:56.112155 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.111787 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" event={"ID":"29baf82c-6242-44a4-a27d-e4c7a4bd24c8","Type":"ContainerDied","Data":"b35a165ceceb6f1ab05ab095522ef97b49ee5578d92c963b68975ff18abd48cb"} Nov 01 01:27:56.249984 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.249950 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:27:56.341913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.341827 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls\") pod \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " Nov 01 01:27:56.342074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.341942 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-metrics-certs\") pod \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " Nov 01 01:27:56.342074 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.342053 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-openshift-service-ca\") pod \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " Nov 01 01:27:56.342165 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.342124 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-kube-api-access-mjfsg\" (UniqueName: \"kubernetes.io/projected/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-kube-api-access-mjfsg\") pod \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\" (UID: \"29baf82c-6242-44a4-a27d-e4c7a4bd24c8\") " Nov 01 01:27:56.342418 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.342387 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-openshift-service-ca" pod "29baf82c-6242-44a4-a27d-e4c7a4bd24c8" (UID: "29baf82c-6242-44a4-a27d-e4c7a4bd24c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:27:56.342587 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.342569 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:56.344200 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.344175 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls" pod "29baf82c-6242-44a4-a27d-e4c7a4bd24c8" (UID: "29baf82c-6242-44a4-a27d-e4c7a4bd24c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:27:56.344352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.344331 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-metrics-certs" pod "29baf82c-6242-44a4-a27d-e4c7a4bd24c8" (UID: "29baf82c-6242-44a4-a27d-e4c7a4bd24c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:27:56.344442 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.344423 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-kube-api-access-mjfsg" pod "29baf82c-6242-44a4-a27d-e4c7a4bd24c8" (UID: "29baf82c-6242-44a4-a27d-e4c7a4bd24c8"). InnerVolumeSpecName "kube-api-access-mjfsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:27:56.443931 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.443894 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjfsg\" (UniqueName: \"kubernetes.io/projected/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-kube-api-access-mjfsg\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:56.443931 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.443928 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:56.443931 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.443938 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29baf82c-6242-44a4-a27d-e4c7a4bd24c8-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:56.670783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.670692 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:56.941110 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:56.941023 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:57.117998 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:57.117969 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" Nov 01 01:27:57.117998 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:57.117984 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr" event={"ID":"29baf82c-6242-44a4-a27d-e4c7a4bd24c8","Type":"ContainerDied","Data":"f23466b7e38ad4a9f471668244feed2ff81e97f32d1ff8aacb1255223aa5bc33"} Nov 01 01:27:57.118528 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:57.118037 6036 scope.go:122] "RemoveContainer" containerID="b35a165ceceb6f1ab05ab095522ef97b49ee5578d92c963b68975ff18abd48cb" Nov 01 01:27:57.284851 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:57.284820 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr"] Nov 01 01:27:57.334840 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:57.334806 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-43c48b-68b496c994-xqlpr"] Nov 01 01:27:57.763005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:57.762901 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:27:58.617006 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:58.616969 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" podUID="5343d36c-a98b-4bfe-a118-c318d6c7d707" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:27:58.617503 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:58.617021 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" podUID="0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:27:59.250899 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:27:59.250856 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29baf82c-6242-44a4-a27d-e4c7a4bd24c8" path="/var/lib/kubelet/pods/29baf82c-6242-44a4-a27d-e4c7a4bd24c8/volumes" Nov 01 01:28:03.767334 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:03.767299 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:28:04.038767 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:04.038731 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:28:04.261334 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:04.261305 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:28:08.617142 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:08.617097 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" podUID="5343d36c-a98b-4bfe-a118-c318d6c7d707" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:28:08.617580 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:08.617155 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" podUID="0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:28:14.020463 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.020394 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:28:14.030559 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.030530 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5343d36c-a98b-4bfe-a118-c318d6c7d707-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5343d36c-a98b-4bfe-a118-c318d6c7d707-plugins-config-volume\") pod \"5343d36c-a98b-4bfe-a118-c318d6c7d707\" (UID: \"5343d36c-a98b-4bfe-a118-c318d6c7d707\") " Nov 01 01:28:14.030685 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.030669 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5343d36c-a98b-4bfe-a118-c318d6c7d707-kube-api-access-9cf5q\" (UniqueName: \"kubernetes.io/projected/5343d36c-a98b-4bfe-a118-c318d6c7d707-kube-api-access-9cf5q\") pod \"5343d36c-a98b-4bfe-a118-c318d6c7d707\" (UID: \"5343d36c-a98b-4bfe-a118-c318d6c7d707\") " Nov 01 01:28:14.031883 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.031841 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5343d36c-a98b-4bfe-a118-c318d6c7d707-plugins-config-volume" pod "5343d36c-a98b-4bfe-a118-c318d6c7d707" (UID: "5343d36c-a98b-4bfe-a118-c318d6c7d707"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:28:14.034742 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.034712 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5343d36c-a98b-4bfe-a118-c318d6c7d707-kube-api-access-9cf5q" pod "5343d36c-a98b-4bfe-a118-c318d6c7d707" (UID: "5343d36c-a98b-4bfe-a118-c318d6c7d707"). InnerVolumeSpecName "kube-api-access-9cf5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:28:14.086853 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.086826 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:28:14.131648 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.131621 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-kube-api-access-782ss\" (UniqueName: \"kubernetes.io/projected/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-kube-api-access-782ss\") pod \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\" (UID: \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\") " Nov 01 01:28:14.131817 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.131701 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-plugins-config-volume\") pod \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\" (UID: \"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4\") " Nov 01 01:28:14.132051 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.132030 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-plugins-config-volume" pod "0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" (UID: "0eb12ab6-f69a-46c2-86e9-cef7cb829fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:28:14.132144 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.132132 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cf5q\" (UniqueName: \"kubernetes.io/projected/5343d36c-a98b-4bfe-a118-c318d6c7d707-kube-api-access-9cf5q\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:28:14.132186 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.132146 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5343d36c-a98b-4bfe-a118-c318d6c7d707-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:28:14.132186 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.132157 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:28:14.134008 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.133989 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-kube-api-access-782ss" pod "0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" (UID: "0eb12ab6-f69a-46c2-86e9-cef7cb829fb4"). InnerVolumeSpecName "kube-api-access-782ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:28:14.211787 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.211747 6036 generic.go:356] "Generic (PLEG): container finished" podID="5343d36c-a98b-4bfe-a118-c318d6c7d707" containerID="5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93" exitCode=0 Nov 01 01:28:14.211983 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.211802 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" Nov 01 01:28:14.211983 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.211835 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" event={"ID":"5343d36c-a98b-4bfe-a118-c318d6c7d707","Type":"ContainerDied","Data":"5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93"} Nov 01 01:28:14.211983 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.211922 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf" event={"ID":"5343d36c-a98b-4bfe-a118-c318d6c7d707","Type":"ContainerDied","Data":"58535f6c1e4368a7cd5639791377e2d66aeefc3ad6ee1735275f3ffb9831836a"} Nov 01 01:28:14.211983 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.211945 6036 scope.go:122] "RemoveContainer" containerID="5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93" Nov 01 01:28:14.213205 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.213177 6036 generic.go:356] "Generic (PLEG): container finished" podID="0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" containerID="251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3" exitCode=0 Nov 01 01:28:14.213325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.213218 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" event={"ID":"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4","Type":"ContainerDied","Data":"251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3"} Nov 01 01:28:14.213325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.213231 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" Nov 01 01:28:14.213325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.213265 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7" event={"ID":"0eb12ab6-f69a-46c2-86e9-cef7cb829fb4","Type":"ContainerDied","Data":"57674c0a84bec6f1fc2185dac156f02bc0c467912d5cb05baead2db064f68d00"} Nov 01 01:28:14.222004 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.221988 6036 scope.go:122] "RemoveContainer" containerID="5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93" Nov 01 01:28:14.222360 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:28:14.222312 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93\": container with ID starting with 5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93 not found: ID does not exist" containerID="5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93" Nov 01 01:28:14.222470 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.222356 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93"} err="failed to get container status \"5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93\": rpc error: code = NotFound desc = could not find container \"5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93\": container with ID starting with 5102a3653c787346a31cc2cb2549d429d7af477d9300a36d05f2db2c7f840c93 not found: ID does not exist" Nov 01 01:28:14.222470 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.222381 6036 scope.go:122] "RemoveContainer" containerID="251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3" Nov 01 01:28:14.230693 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.230676 6036 scope.go:122] "RemoveContainer" containerID="251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3" Nov 01 01:28:14.230928 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:28:14.230910 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3\": container with ID starting with 251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3 not found: ID does not exist" containerID="251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3" Nov 01 01:28:14.231001 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.230937 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3"} err="failed to get container status \"251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3\": rpc error: code = NotFound desc = could not find container \"251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3\": container with ID starting with 251851d33db2f2ee009e5337b672783445115cf7ff12c5010bcdf3746dbaf1b3 not found: ID does not exist" Nov 01 01:28:14.233491 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.233474 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-782ss\" (UniqueName: \"kubernetes.io/projected/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4-kube-api-access-782ss\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:28:14.369047 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.369010 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf"] Nov 01 01:28:14.373073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.373047 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-43c48b-686c66db6bg6xf"] Nov 01 01:28:14.484516 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.484483 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7"] Nov 01 01:28:14.488923 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:14.488894 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-43c48b-6676f7xrs7"] Nov 01 01:28:15.249965 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:15.249923 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" path="/var/lib/kubelet/pods/0eb12ab6-f69a-46c2-86e9-cef7cb829fb4/volumes" Nov 01 01:28:15.250476 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:15.250336 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5343d36c-a98b-4bfe-a118-c318d6c7d707" path="/var/lib/kubelet/pods/5343d36c-a98b-4bfe-a118-c318d6c7d707/volumes" Nov 01 01:28:18.577772 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:18.577741 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:28:19.088100 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:19.088060 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:28:19.504423 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:28:19.504338 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:28:24.666664 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:24.666633 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:28:42.770519 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:28:42.770487 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:28:49.504017 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:28:49.503981 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:29:19.505076 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:29:19.505038 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:29:43.117611 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:43.117577 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:29:49.504117 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:29:49.504078 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:29:50.012142 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.012113 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:29:50.051721 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.051684 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l"] Nov 01 01:29:50.052228 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052198 6036 cpu_manager.go:414] "containerMap: removing container" podUID="29baf82c-6242-44a4-a27d-e4c7a4bd24c8" containerName="maas-api" Nov 01 01:29:50.052228 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052222 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29baf82c-6242-44a4-a27d-e4c7a4bd24c8" containerName="maas-api" Nov 01 01:29:50.052482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052261 6036 cpu_manager.go:414] "containerMap: removing container" podUID="5343d36c-a98b-4bfe-a118-c318d6c7d707" containerName="payload-processing" Nov 01 01:29:50.052482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052269 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5343d36c-a98b-4bfe-a118-c318d6c7d707" containerName="payload-processing" Nov 01 01:29:50.052482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052314 6036 cpu_manager.go:414] "containerMap: removing container" podUID="0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" containerName="payload-pre-processing" Nov 01 01:29:50.052482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052322 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" containerName="payload-pre-processing" Nov 01 01:29:50.052482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052417 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb12ab6-f69a-46c2-86e9-cef7cb829fb4" containerName="payload-pre-processing" Nov 01 01:29:50.052482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052437 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="29baf82c-6242-44a4-a27d-e4c7a4bd24c8" containerName="maas-api" Nov 01 01:29:50.052482 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.052444 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5343d36c-a98b-4bfe-a118-c318d6c7d707" containerName="payload-processing" Nov 01 01:29:50.055750 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.055732 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.161445 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.161412 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gp6r7\"" Nov 01 01:29:50.161665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.161478 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 01 01:29:50.161665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.161576 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 01:29:50.161665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.161629 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 01:29:50.161665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.161634 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 01 01:29:50.163269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.163248 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l"] Nov 01 01:29:50.216954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.216918 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgfvg\" (UniqueName: \"kubernetes.io/projected/2826f180-d22f-4ab1-8675-95d4f6b91882-kube-api-access-wgfvg\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.217129 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.216969 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2826f180-d22f-4ab1-8675-95d4f6b91882-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.217129 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.217047 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2826f180-d22f-4ab1-8675-95d4f6b91882-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.238791 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.238754 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:29:50.317664 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.317625 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgfvg\" (UniqueName: \"kubernetes.io/projected/2826f180-d22f-4ab1-8675-95d4f6b91882-kube-api-access-wgfvg\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.317876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.317690 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2826f180-d22f-4ab1-8675-95d4f6b91882-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.317876 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.317736 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2826f180-d22f-4ab1-8675-95d4f6b91882-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.318541 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.318512 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2826f180-d22f-4ab1-8675-95d4f6b91882-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.320404 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.320383 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2826f180-d22f-4ab1-8675-95d4f6b91882-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.380247 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.380213 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgfvg\" (UniqueName: \"kubernetes.io/projected/2826f180-d22f-4ab1-8675-95d4f6b91882-kube-api-access-wgfvg\") pod \"kuadrant-console-plugin-66c4fb575d-l495l\" (UID: \"2826f180-d22f-4ab1-8675-95d4f6b91882\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:50.668061 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:50.667962 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" Nov 01 01:29:51.059128 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:51.059093 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l"] Nov 01 01:29:51.062992 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:29:51.062950 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2826f180_d22f_4ab1_8675_95d4f6b91882.slice/crio-c9c785c0b376b2e813cdb8737d61664e94fb081e92f72590eba7c7add8b2a293 WatchSource:0}: Error finding container c9c785c0b376b2e813cdb8737d61664e94fb081e92f72590eba7c7add8b2a293: Status 404 returned error can't find the container with id c9c785c0b376b2e813cdb8737d61664e94fb081e92f72590eba7c7add8b2a293 Nov 01 01:29:51.704697 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:51.704655 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" event={"ID":"2826f180-d22f-4ab1-8675-95d4f6b91882","Type":"ContainerStarted","Data":"c9c785c0b376b2e813cdb8737d61664e94fb081e92f72590eba7c7add8b2a293"} Nov 01 01:29:52.898298 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:52.898184 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:29:53.121634 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:53.121598 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:29:57.751068 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:57.751026 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" event={"ID":"2826f180-d22f-4ab1-8675-95d4f6b91882","Type":"ContainerStarted","Data":"b19d648957450dd694a71971df6c6f3f86a508baa789c4655dd9fc26f377a310"} Nov 01 01:29:57.871803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:29:57.871750 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l495l" podStartSLOduration=1.7629024279999999 podStartE2EDuration="7.871732472s" totalImagesPullingTime="6.108830044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:29:57.870910714 +0000 UTC m=+3942.746131729" watchObservedRunningTime="2026-11-01 01:29:57.871732472 +0000 UTC m=+3942.746953562" Nov 01 01:30:19.505816 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:30:19.505775 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:30:36.487551 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:36.487465 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:37.148002 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:37.147962 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:37.360623 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:37.360585 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:41.483819 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:41.483782 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:41.706652 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:41.706616 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:43.409496 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:43.409463 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:44.231059 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:44.231025 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:44.439654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:44.439616 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:44.719744 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:44.719713 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:44.930898 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:44.930781 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:45.061951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.061916 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj"] Nov 01 01:30:45.080681 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.080648 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:45.123755 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.123718 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj"] Nov 01 01:30:45.153868 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.153827 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1d30-402e-4c71-8447-5941b7f83087-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj\" (UID: \"2e4d1d30-402e-4c71-8447-5941b7f83087\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:45.154078 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.153977 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957nh\" (UniqueName: \"kubernetes.io/projected/2e4d1d30-402e-4c71-8447-5941b7f83087-kube-api-access-957nh\") pod \"payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj\" (UID: \"2e4d1d30-402e-4c71-8447-5941b7f83087\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:45.172938 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.172884 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1048fc\"" Nov 01 01:30:45.173107 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.172940 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8e2b465b-dockercfg-5mpvs\"" Nov 01 01:30:45.225338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.225230 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz"] Nov 01 01:30:45.229176 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.229154 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.254991 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.254960 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957nh\" (UniqueName: \"kubernetes.io/projected/2e4d1d30-402e-4c71-8447-5941b7f83087-kube-api-access-957nh\") pod \"payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj\" (UID: \"2e4d1d30-402e-4c71-8447-5941b7f83087\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:45.255147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.255076 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a63d2a-9a99-434a-ad27-1e56807e1200-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz\" (UID: \"d6a63d2a-9a99-434a-ad27-1e56807e1200\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.255147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.255112 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlvz\" (UniqueName: \"kubernetes.io/projected/d6a63d2a-9a99-434a-ad27-1e56807e1200-kube-api-access-4rlvz\") pod \"payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz\" (UID: \"d6a63d2a-9a99-434a-ad27-1e56807e1200\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.255224 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.255165 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1d30-402e-4c71-8447-5941b7f83087-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj\" (UID: \"2e4d1d30-402e-4c71-8447-5941b7f83087\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:45.255857 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.255836 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1d30-402e-4c71-8447-5941b7f83087-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj\" (UID: \"2e4d1d30-402e-4c71-8447-5941b7f83087\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:45.281939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.281900 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz"] Nov 01 01:30:45.317093 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.317064 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-957nh\" (UniqueName: \"kubernetes.io/projected/2e4d1d30-402e-4c71-8447-5941b7f83087-kube-api-access-957nh\") pod \"payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj\" (UID: \"2e4d1d30-402e-4c71-8447-5941b7f83087\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:45.356411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.356376 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a63d2a-9a99-434a-ad27-1e56807e1200-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz\" (UID: \"d6a63d2a-9a99-434a-ad27-1e56807e1200\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.356411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.356415 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlvz\" (UniqueName: \"kubernetes.io/projected/d6a63d2a-9a99-434a-ad27-1e56807e1200-kube-api-access-4rlvz\") pod \"payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz\" (UID: \"d6a63d2a-9a99-434a-ad27-1e56807e1200\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.357152 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.357135 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a63d2a-9a99-434a-ad27-1e56807e1200-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz\" (UID: \"d6a63d2a-9a99-434a-ad27-1e56807e1200\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.394249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.394202 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:45.419024 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.418984 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlvz\" (UniqueName: \"kubernetes.io/projected/d6a63d2a-9a99-434a-ad27-1e56807e1200-kube-api-access-4rlvz\") pod \"payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz\" (UID: \"d6a63d2a-9a99-434a-ad27-1e56807e1200\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.542668 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.542634 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.768811 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.768785 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj"] Nov 01 01:30:45.770827 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:30:45.770800 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4d1d30_402e_4c71_8447_5941b7f83087.slice/crio-5a11472c2d9915f065476b3e11ec510862f53b2adc7976b999456c27dfa324ea WatchSource:0}: Error finding container 5a11472c2d9915f065476b3e11ec510862f53b2adc7976b999456c27dfa324ea: Status 404 returned error can't find the container with id 5a11472c2d9915f065476b3e11ec510862f53b2adc7976b999456c27dfa324ea Nov 01 01:30:45.914825 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.914799 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz"] Nov 01 01:30:45.917540 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:30:45.917510 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a63d2a_9a99_434a_ad27_1e56807e1200.slice/crio-bcf2b59bb81a0fdb8562c9ac275e596c0354f4fea56e208df4fcc15fed4c5d88 WatchSource:0}: Error finding container bcf2b59bb81a0fdb8562c9ac275e596c0354f4fea56e208df4fcc15fed4c5d88: Status 404 returned error can't find the container with id bcf2b59bb81a0fdb8562c9ac275e596c0354f4fea56e208df4fcc15fed4c5d88 Nov 01 01:30:45.996971 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.996933 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" event={"ID":"d6a63d2a-9a99-434a-ad27-1e56807e1200","Type":"ContainerStarted","Data":"3966537f8901f843c0697dc8ff192f6453af5aec4949ba50e79ce96a8c577824"} Nov 01 01:30:45.997147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.997002 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" event={"ID":"d6a63d2a-9a99-434a-ad27-1e56807e1200","Type":"ContainerStarted","Data":"bcf2b59bb81a0fdb8562c9ac275e596c0354f4fea56e208df4fcc15fed4c5d88"} Nov 01 01:30:45.997147 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.997093 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:45.998786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.998764 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" event={"ID":"2e4d1d30-402e-4c71-8447-5941b7f83087","Type":"ContainerStarted","Data":"8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59"} Nov 01 01:30:45.998894 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.998801 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" event={"ID":"2e4d1d30-402e-4c71-8447-5941b7f83087","Type":"ContainerStarted","Data":"5a11472c2d9915f065476b3e11ec510862f53b2adc7976b999456c27dfa324ea"} Nov 01 01:30:45.998894 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:45.998870 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:46.115893 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:46.115779 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" podStartSLOduration=1.115764129 podStartE2EDuration="1.115764129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:30:46.114169493 +0000 UTC m=+3990.989390521" watchObservedRunningTime="2026-11-01 01:30:46.115764129 +0000 UTC m=+3990.990985143" Nov 01 01:30:46.232769 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:46.232702 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" podStartSLOduration=1.232683591 podStartE2EDuration="1.232683591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:30:46.230899922 +0000 UTC m=+3991.106120971" watchObservedRunningTime="2026-11-01 01:30:46.232683591 +0000 UTC m=+3991.107904607" Nov 01 01:30:47.423101 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.423066 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql"] Nov 01 01:30:47.426956 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.426923 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.471875 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.471839 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e2b654-93a9-4f94-b2ad-4055be914665-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.472072 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.471903 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.472072 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.472049 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.472177 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.472150 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpf4s\" (UniqueName: \"kubernetes.io/projected/d8e2b654-93a9-4f94-b2ad-4055be914665-kube-api-access-jpf4s\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.480073 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.480047 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-1048fc-serving-cert\"" Nov 01 01:30:47.487305 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.487263 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql"] Nov 01 01:30:47.572783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.572749 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e2b654-93a9-4f94-b2ad-4055be914665-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.572952 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.572803 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.572952 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.572872 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.572952 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.572940 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpf4s\" (UniqueName: \"kubernetes.io/projected/d8e2b654-93a9-4f94-b2ad-4055be914665-kube-api-access-jpf4s\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.573154 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:30:47.572973 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-serving-cert: secret "maas-api-e2e-worker-single-main-1048fc-serving-cert" not found Nov 01 01:30:47.573154 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:30:47.573040 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls podName:d8e2b654-93a9-4f94-b2ad-4055be914665 nodeName:}" failed. No retries permitted until 2026-11-01 01:30:48.073020135 +0000 UTC m=+3992.948241153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls") pod "maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" (UID: "d8e2b654-93a9-4f94-b2ad-4055be914665") : secret "maas-api-e2e-worker-single-main-1048fc-serving-cert" not found Nov 01 01:30:47.573527 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.573503 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e2b654-93a9-4f94-b2ad-4055be914665-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.575802 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.575778 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.636053 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.636014 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpf4s\" (UniqueName: \"kubernetes.io/projected/d8e2b654-93a9-4f94-b2ad-4055be914665-kube-api-access-jpf4s\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:47.693921 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.693820 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:47.907522 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:47.907486 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:48.078608 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:48.078573 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:48.081446 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:48.081410 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:48.120602 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:48.120554 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:48.340697 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:48.340609 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:48.717724 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:48.717469 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql"] Nov 01 01:30:48.720004 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:30:48.719974 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e2b654_93a9_4f94_b2ad_4055be914665.slice/crio-a442d2c4f2ead803bdf55e88decc6e1247033b8f105cedffd94275e0aa7ec2c6 WatchSource:0}: Error finding container a442d2c4f2ead803bdf55e88decc6e1247033b8f105cedffd94275e0aa7ec2c6: Status 404 returned error can't find the container with id a442d2c4f2ead803bdf55e88decc6e1247033b8f105cedffd94275e0aa7ec2c6 Nov 01 01:30:49.016939 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:49.016858 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" event={"ID":"d8e2b654-93a9-4f94-b2ad-4055be914665","Type":"ContainerStarted","Data":"a442d2c4f2ead803bdf55e88decc6e1247033b8f105cedffd94275e0aa7ec2c6"} Nov 01 01:30:49.504504 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:30:49.504471 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:30:50.023603 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:50.023559 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" event={"ID":"d8e2b654-93a9-4f94-b2ad-4055be914665","Type":"ContainerStarted","Data":"f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3"} Nov 01 01:30:50.024091 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:50.023643 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:50.143759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:50.143706 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" podStartSLOduration=2.199221105 podStartE2EDuration="3.143687934s" totalImagesPullingTime="944.466829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:30:50.142895619 +0000 UTC m=+3995.018116634" watchObservedRunningTime="2026-11-01 01:30:50.143687934 +0000 UTC m=+3995.018908981" Nov 01 01:30:52.730897 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:52.730863 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:30:56.034649 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:56.034618 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:30:57.007126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:57.007094 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:30:57.007325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:57.007162 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:30:57.637398 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:30:57.637361 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:02.436005 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:02.435962 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:05.588770 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:05.588734 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:06.420733 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:06.420698 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:07.006566 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:07.006534 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:07.220515 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:07.220477 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:07.537559 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:07.537529 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:09.601625 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:09.601593 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:12.239095 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:12.239065 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:17.046043 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:17.046010 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:19.504870 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:31:19.504821 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:31:21.846691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:21.846659 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:26.651053 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:26.651020 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:31.433804 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:31.433758 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:36.328055 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:36.328020 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:41.180697 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:41.180659 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:43.755855 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:43.755826 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:45.945616 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:45.945583 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:48.104652 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:48.104615 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:49.505757 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:31:49.505718 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:31:50.739803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:50.739767 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:31:55.630259 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:31:55.630172 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:00.440269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:00.440232 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:05.323620 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:05.323579 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:06.494578 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:06.494540 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:10.593524 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:10.593486 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:11.527803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:11.527769 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:15.630961 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:15.630916 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:18.487593 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:18.487555 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz"] Nov 01 01:32:18.582720 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:18.582682 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj"] Nov 01 01:32:18.917675 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:18.917639 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:19.131112 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:19.131075 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:19.505213 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:32:19.505178 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:32:20.443464 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:20.443426 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:23.488362 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:23.488318 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" podUID="d6a63d2a-9a99-434a-ad27-1e56807e1200" containerName="payload-processing" containerID="cri-o://3966537f8901f843c0697dc8ff192f6453af5aec4949ba50e79ce96a8c577824" gracePeriod=25 Nov 01 01:32:23.583656 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:23.583612 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" podUID="2e4d1d30-402e-4c71-8447-5941b7f83087" containerName="payload-pre-processing" containerID="cri-o://8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59" gracePeriod=25 Nov 01 01:32:24.489483 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:24.489444 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql"] Nov 01 01:32:24.489992 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:24.489710 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" podUID="d8e2b654-93a9-4f94-b2ad-4055be914665" containerName="maas-api" containerID="cri-o://f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3" gracePeriod=30 Nov 01 01:32:24.687036 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:24.687000 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:25.750003 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.749972 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:32:25.835578 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.835493 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls\") pod \"d8e2b654-93a9-4f94-b2ad-4055be914665\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " Nov 01 01:32:25.835753 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.835643 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-metrics-certs\") pod \"d8e2b654-93a9-4f94-b2ad-4055be914665\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " Nov 01 01:32:25.835753 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.835708 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e2b654-93a9-4f94-b2ad-4055be914665-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e2b654-93a9-4f94-b2ad-4055be914665-openshift-service-ca\") pod \"d8e2b654-93a9-4f94-b2ad-4055be914665\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " Nov 01 01:32:25.835753 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.835742 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e2b654-93a9-4f94-b2ad-4055be914665-kube-api-access-jpf4s\" (UniqueName: \"kubernetes.io/projected/d8e2b654-93a9-4f94-b2ad-4055be914665-kube-api-access-jpf4s\") pod \"d8e2b654-93a9-4f94-b2ad-4055be914665\" (UID: \"d8e2b654-93a9-4f94-b2ad-4055be914665\") " Nov 01 01:32:25.836079 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.836050 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e2b654-93a9-4f94-b2ad-4055be914665-openshift-service-ca" pod "d8e2b654-93a9-4f94-b2ad-4055be914665" (UID: "d8e2b654-93a9-4f94-b2ad-4055be914665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:32:25.837820 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.837794 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-metrics-certs" pod "d8e2b654-93a9-4f94-b2ad-4055be914665" (UID: "d8e2b654-93a9-4f94-b2ad-4055be914665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:32:25.837922 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.837872 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls" pod "d8e2b654-93a9-4f94-b2ad-4055be914665" (UID: "d8e2b654-93a9-4f94-b2ad-4055be914665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:32:25.838039 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.838020 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e2b654-93a9-4f94-b2ad-4055be914665-kube-api-access-jpf4s" pod "d8e2b654-93a9-4f94-b2ad-4055be914665" (UID: "d8e2b654-93a9-4f94-b2ad-4055be914665"). InnerVolumeSpecName "kube-api-access-jpf4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:32:25.936837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.936795 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:25.936837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.936830 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e2b654-93a9-4f94-b2ad-4055be914665-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:25.936837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.936839 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpf4s\" (UniqueName: \"kubernetes.io/projected/d8e2b654-93a9-4f94-b2ad-4055be914665-kube-api-access-jpf4s\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:25.936837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:25.936849 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e2b654-93a9-4f94-b2ad-4055be914665-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:26.174895 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.174812 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:26.444764 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.444682 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:26.516883 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.516846 6036 generic.go:356] "Generic (PLEG): container finished" podID="d8e2b654-93a9-4f94-b2ad-4055be914665" containerID="f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3" exitCode=0 Nov 01 01:32:26.517159 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.516893 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" Nov 01 01:32:26.517159 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.516932 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" event={"ID":"d8e2b654-93a9-4f94-b2ad-4055be914665","Type":"ContainerDied","Data":"f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3"} Nov 01 01:32:26.517159 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.516987 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql" event={"ID":"d8e2b654-93a9-4f94-b2ad-4055be914665","Type":"ContainerDied","Data":"a442d2c4f2ead803bdf55e88decc6e1247033b8f105cedffd94275e0aa7ec2c6"} Nov 01 01:32:26.517159 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.517010 6036 scope.go:122] "RemoveContainer" containerID="f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3" Nov 01 01:32:26.527018 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.526993 6036 scope.go:122] "RemoveContainer" containerID="f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3" Nov 01 01:32:26.527310 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:32:26.527265 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3\": container with ID starting with f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3 not found: ID does not exist" containerID="f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3" Nov 01 01:32:26.527390 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.527319 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3"} err="failed to get container status \"f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3\": rpc error: code = NotFound desc = could not find container \"f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3\": container with ID starting with f80404289695cad396f2c0e4356a7503bdd923147889724414eed12c5d009bb3 not found: ID does not exist" Nov 01 01:32:26.685730 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.685696 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql"] Nov 01 01:32:26.736249 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:26.736165 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1048fc-79dfc7dc4d-lhnql"] Nov 01 01:32:27.005523 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:27.005425 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" podUID="d6a63d2a-9a99-434a-ad27-1e56807e1200" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:32:27.005523 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:27.005453 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" podUID="2e4d1d30-402e-4c71-8447-5941b7f83087" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:32:27.220245 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:27.220209 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:27.249076 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:27.249044 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e2b654-93a9-4f94-b2ad-4055be914665" path="/var/lib/kubelet/pods/d8e2b654-93a9-4f94-b2ad-4055be914665/volumes" Nov 01 01:32:33.056815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:33.056783 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:33.267783 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:33.267749 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:33.519378 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:33.519270 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:37.005621 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:37.005581 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" podUID="d6a63d2a-9a99-434a-ad27-1e56807e1200" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:32:37.006111 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:37.005623 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" podUID="2e4d1d30-402e-4c71-8447-5941b7f83087" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:32:40.609017 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:40.608984 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:43.612371 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:43.612338 6036 generic.go:356] "Generic (PLEG): container finished" podID="d6a63d2a-9a99-434a-ad27-1e56807e1200" containerID="3966537f8901f843c0697dc8ff192f6453af5aec4949ba50e79ce96a8c577824" exitCode=0 Nov 01 01:32:43.612753 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:43.612406 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" event={"ID":"d6a63d2a-9a99-434a-ad27-1e56807e1200","Type":"ContainerDied","Data":"3966537f8901f843c0697dc8ff192f6453af5aec4949ba50e79ce96a8c577824"} Nov 01 01:32:43.854828 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:43.854799 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:32:43.874258 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:43.874232 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:32:44.037775 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.037740 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a63d2a-9a99-434a-ad27-1e56807e1200-kube-api-access-4rlvz\" (UniqueName: \"kubernetes.io/projected/d6a63d2a-9a99-434a-ad27-1e56807e1200-kube-api-access-4rlvz\") pod \"d6a63d2a-9a99-434a-ad27-1e56807e1200\" (UID: \"d6a63d2a-9a99-434a-ad27-1e56807e1200\") " Nov 01 01:32:44.037977 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.037849 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4d1d30-402e-4c71-8447-5941b7f83087-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1d30-402e-4c71-8447-5941b7f83087-plugins-config-volume\") pod \"2e4d1d30-402e-4c71-8447-5941b7f83087\" (UID: \"2e4d1d30-402e-4c71-8447-5941b7f83087\") " Nov 01 01:32:44.037977 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.037898 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4d1d30-402e-4c71-8447-5941b7f83087-kube-api-access-957nh\" (UniqueName: \"kubernetes.io/projected/2e4d1d30-402e-4c71-8447-5941b7f83087-kube-api-access-957nh\") pod \"2e4d1d30-402e-4c71-8447-5941b7f83087\" (UID: \"2e4d1d30-402e-4c71-8447-5941b7f83087\") " Nov 01 01:32:44.037977 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.037942 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a63d2a-9a99-434a-ad27-1e56807e1200-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a63d2a-9a99-434a-ad27-1e56807e1200-plugins-config-volume\") pod \"d6a63d2a-9a99-434a-ad27-1e56807e1200\" (UID: \"d6a63d2a-9a99-434a-ad27-1e56807e1200\") " Nov 01 01:32:44.038351 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.038320 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4d1d30-402e-4c71-8447-5941b7f83087-plugins-config-volume" pod "2e4d1d30-402e-4c71-8447-5941b7f83087" (UID: "2e4d1d30-402e-4c71-8447-5941b7f83087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:32:44.038463 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.038389 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a63d2a-9a99-434a-ad27-1e56807e1200-plugins-config-volume" pod "d6a63d2a-9a99-434a-ad27-1e56807e1200" (UID: "d6a63d2a-9a99-434a-ad27-1e56807e1200"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:32:44.038587 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.038462 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1d30-402e-4c71-8447-5941b7f83087-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:44.038587 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.038481 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a63d2a-9a99-434a-ad27-1e56807e1200-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:44.040252 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.040231 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4d1d30-402e-4c71-8447-5941b7f83087-kube-api-access-957nh" pod "2e4d1d30-402e-4c71-8447-5941b7f83087" (UID: "2e4d1d30-402e-4c71-8447-5941b7f83087"). InnerVolumeSpecName "kube-api-access-957nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:32:44.040252 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.040240 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a63d2a-9a99-434a-ad27-1e56807e1200-kube-api-access-4rlvz" pod "d6a63d2a-9a99-434a-ad27-1e56807e1200" (UID: "d6a63d2a-9a99-434a-ad27-1e56807e1200"). InnerVolumeSpecName "kube-api-access-4rlvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:32:44.109882 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.109850 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:44.139852 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.139817 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rlvz\" (UniqueName: \"kubernetes.io/projected/d6a63d2a-9a99-434a-ad27-1e56807e1200-kube-api-access-4rlvz\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:44.139852 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.139844 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957nh\" (UniqueName: \"kubernetes.io/projected/2e4d1d30-402e-4c71-8447-5941b7f83087-kube-api-access-957nh\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:44.621868 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.621828 6036 generic.go:356] "Generic (PLEG): container finished" podID="2e4d1d30-402e-4c71-8447-5941b7f83087" containerID="8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59" exitCode=0 Nov 01 01:32:44.622349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.621880 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" Nov 01 01:32:44.622349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.621907 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" event={"ID":"2e4d1d30-402e-4c71-8447-5941b7f83087","Type":"ContainerDied","Data":"8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59"} Nov 01 01:32:44.622349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.621972 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj" event={"ID":"2e4d1d30-402e-4c71-8447-5941b7f83087","Type":"ContainerDied","Data":"5a11472c2d9915f065476b3e11ec510862f53b2adc7976b999456c27dfa324ea"} Nov 01 01:32:44.622349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.622001 6036 scope.go:122] "RemoveContainer" containerID="8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59" Nov 01 01:32:44.623405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.623380 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" event={"ID":"d6a63d2a-9a99-434a-ad27-1e56807e1200","Type":"ContainerDied","Data":"bcf2b59bb81a0fdb8562c9ac275e596c0354f4fea56e208df4fcc15fed4c5d88"} Nov 01 01:32:44.623405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.623400 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz" Nov 01 01:32:44.632375 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.632352 6036 scope.go:122] "RemoveContainer" containerID="8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59" Nov 01 01:32:44.632681 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:32:44.632656 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59\": container with ID starting with 8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59 not found: ID does not exist" containerID="8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59" Nov 01 01:32:44.632747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.632691 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59"} err="failed to get container status \"8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59\": rpc error: code = NotFound desc = could not find container \"8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59\": container with ID starting with 8aa97d2a118a3c657291163b669065f22092c0adb9aa584d1b7848f08ed2fe59 not found: ID does not exist" Nov 01 01:32:44.632747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.632707 6036 scope.go:122] "RemoveContainer" containerID="3966537f8901f843c0697dc8ff192f6453af5aec4949ba50e79ce96a8c577824" Nov 01 01:32:44.788172 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.788140 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj"] Nov 01 01:32:44.838468 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.838432 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1048fc-696b7jb6rj"] Nov 01 01:32:44.951003 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.950963 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz"] Nov 01 01:32:44.955815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:44.955787 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1048fc-7788b79b4rrksz"] Nov 01 01:32:45.250069 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:45.249990 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4d1d30-402e-4c71-8447-5941b7f83087" path="/var/lib/kubelet/pods/2e4d1d30-402e-4c71-8447-5941b7f83087/volumes" Nov 01 01:32:45.250446 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:45.250431 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a63d2a-9a99-434a-ad27-1e56807e1200" path="/var/lib/kubelet/pods/d6a63d2a-9a99-434a-ad27-1e56807e1200/volumes" Nov 01 01:32:48.985242 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:48.985208 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:49.423682 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:32:49.423648 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:32:49.503710 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:32:49.503674 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:33:04.262366 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:33:04.262331 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:33:19.504456 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:33:19.504418 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:33:22.561085 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:33:22.561047 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:33:37.186399 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:33:37.186364 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:33:44.462468 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:33:44.462428 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:33:49.504264 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:33:49.504218 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:34:10.494843 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:34:10.494810 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:34:19.504327 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:34:19.504266 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:34:29.293036 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:34:29.292997 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:34:36.551195 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:34:36.551150 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 01:34:36.551697 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:34:36.551220 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 01:34:44.867636 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:34:44.867601 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:34:49.504951 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:34:49.504915 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:35:19.504827 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:35:19.504786 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:35:20.483243 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483209 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch"] Nov 01 01:35:20.483754 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483736 6036 cpu_manager.go:414] "containerMap: removing container" podUID="2e4d1d30-402e-4c71-8447-5941b7f83087" containerName="payload-pre-processing" Nov 01 01:35:20.483807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483756 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4d1d30-402e-4c71-8447-5941b7f83087" containerName="payload-pre-processing" Nov 01 01:35:20.483807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483778 6036 cpu_manager.go:414] "containerMap: removing container" podUID="d6a63d2a-9a99-434a-ad27-1e56807e1200" containerName="payload-processing" Nov 01 01:35:20.483807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483783 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a63d2a-9a99-434a-ad27-1e56807e1200" containerName="payload-processing" Nov 01 01:35:20.483807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483791 6036 cpu_manager.go:414] "containerMap: removing container" podUID="d8e2b654-93a9-4f94-b2ad-4055be914665" containerName="maas-api" Nov 01 01:35:20.483807 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483796 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e2b654-93a9-4f94-b2ad-4055be914665" containerName="maas-api" Nov 01 01:35:20.483997 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483875 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4d1d30-402e-4c71-8447-5941b7f83087" containerName="payload-pre-processing" Nov 01 01:35:20.483997 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483886 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a63d2a-9a99-434a-ad27-1e56807e1200" containerName="payload-processing" Nov 01 01:35:20.483997 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.483892 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e2b654-93a9-4f94-b2ad-4055be914665" containerName="maas-api" Nov 01 01:35:20.487280 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.487262 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.595169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.595140 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch"] Nov 01 01:35:20.608948 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.608910 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d9c89-3e83-4240-8565-2e16cbb4c36c-openshift-service-ca\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.609140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.609056 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/771d9c89-3e83-4240-8565-2e16cbb4c36c-metrics-certs\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.609209 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.609135 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5495\" (UniqueName: \"kubernetes.io/projected/771d9c89-3e83-4240-8565-2e16cbb4c36c-kube-api-access-x5495\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.609259 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.609211 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/771d9c89-3e83-4240-8565-2e16cbb4c36c-maas-api-tls\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.710137 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.710099 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/771d9c89-3e83-4240-8565-2e16cbb4c36c-metrics-certs\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.710405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.710161 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5495\" (UniqueName: \"kubernetes.io/projected/771d9c89-3e83-4240-8565-2e16cbb4c36c-kube-api-access-x5495\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.710405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.710198 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/771d9c89-3e83-4240-8565-2e16cbb4c36c-maas-api-tls\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.710405 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.710234 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d9c89-3e83-4240-8565-2e16cbb4c36c-openshift-service-ca\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.710995 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.710915 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d9c89-3e83-4240-8565-2e16cbb4c36c-openshift-service-ca\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.712888 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.712863 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/771d9c89-3e83-4240-8565-2e16cbb4c36c-metrics-certs\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.712983 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.712891 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/771d9c89-3e83-4240-8565-2e16cbb4c36c-maas-api-tls\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.773157 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.773082 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5495\" (UniqueName: \"kubernetes.io/projected/771d9c89-3e83-4240-8565-2e16cbb4c36c-kube-api-access-x5495\") pod \"maas-api-6b97d8bb58-ll7ch\" (UID: \"771d9c89-3e83-4240-8565-2e16cbb4c36c\") " pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:20.801010 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:20.800972 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:21.176540 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:21.176508 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch"] Nov 01 01:35:21.179372 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:35:21.179345 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771d9c89_3e83_4240_8565_2e16cbb4c36c.slice/crio-ac9e4ef28f785f645b69e3100db39021677deea4f060db64eccfe32839418974 WatchSource:0}: Error finding container ac9e4ef28f785f645b69e3100db39021677deea4f060db64eccfe32839418974: Status 404 returned error can't find the container with id ac9e4ef28f785f645b69e3100db39021677deea4f060db64eccfe32839418974 Nov 01 01:35:21.180671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:21.180653 6036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 01:35:21.488716 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:21.488623 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" event={"ID":"771d9c89-3e83-4240-8565-2e16cbb4c36c","Type":"ContainerStarted","Data":"ac9e4ef28f785f645b69e3100db39021677deea4f060db64eccfe32839418974"} Nov 01 01:35:22.495242 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:22.495150 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" event={"ID":"771d9c89-3e83-4240-8565-2e16cbb4c36c","Type":"ContainerStarted","Data":"4b715da2a21589579007e9d3d278ae17734d66f1ca7a3820352e15329f08a79c"} Nov 01 01:35:22.495747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:22.495332 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:22.615273 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:22.615216 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" podStartSLOduration=1.74280623 podStartE2EDuration="2.615202142s" totalImagesPullingTime="872.395912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:35:22.613740276 +0000 UTC m=+4267.488961293" watchObservedRunningTime="2026-11-01 01:35:22.615202142 +0000 UTC m=+4267.490423216" Nov 01 01:35:27.017545 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:27.017513 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:27.795673 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:27.795637 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:28.010453 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:28.010405 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:28.506811 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:28.506782 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b97d8bb58-ll7ch" Nov 01 01:35:31.365239 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:31.365203 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:31.577688 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:31.577655 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:34.597238 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:34.597207 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:34.808929 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:34.808895 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:34.951659 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:34.951571 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p"] Nov 01 01:35:34.955691 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:34.955667 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:35.013380 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.013344 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p"] Nov 01 01:35:35.061520 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.061485 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p\" (UID: \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:35.061693 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.061607 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n596q\" (UniqueName: \"kubernetes.io/projected/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-kube-api-access-n596q\") pod \"payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p\" (UID: \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:35.061871 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.061854 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a684c9cb\"" Nov 01 01:35:35.163126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.163087 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p\" (UID: \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:35.163428 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.163212 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n596q\" (UniqueName: \"kubernetes.io/projected/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-kube-api-access-n596q\") pod \"payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p\" (UID: \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:35.163927 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.163903 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p\" (UID: \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:35.225242 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.225173 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n596q\" (UniqueName: \"kubernetes.io/projected/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-kube-api-access-n596q\") pod \"payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p\" (UID: \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:35.269749 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.269708 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:35.523759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.523675 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9"] Nov 01 01:35:35.528572 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.528547 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.568132 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.568097 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsln5\" (UniqueName: \"kubernetes.io/projected/fadfb67a-6154-4787-8bae-7c29efde99bc-kube-api-access-wsln5\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.568359 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.568157 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.568359 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.568260 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-metrics-certs\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.568466 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.568392 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfb67a-6154-4787-8bae-7c29efde99bc-openshift-service-ca\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.584976 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.584949 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a684c9cb-serving-cert\"" Nov 01 01:35:35.586462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.586437 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9"] Nov 01 01:35:35.643835 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.643807 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p"] Nov 01 01:35:35.645772 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:35:35.645746 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca7b2e2_e635_4fa4_ab3c_ead52ce74d22.slice/crio-fbee8a367a37c6da599a89c697b52ae0863f769a5419f138cf9784370acb4a7f WatchSource:0}: Error finding container fbee8a367a37c6da599a89c697b52ae0863f769a5419f138cf9784370acb4a7f: Status 404 returned error can't find the container with id fbee8a367a37c6da599a89c697b52ae0863f769a5419f138cf9784370acb4a7f Nov 01 01:35:35.669614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.669577 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsln5\" (UniqueName: \"kubernetes.io/projected/fadfb67a-6154-4787-8bae-7c29efde99bc-kube-api-access-wsln5\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.669769 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.669649 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.669769 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.669721 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-metrics-certs\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.669890 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.669787 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfb67a-6154-4787-8bae-7c29efde99bc-openshift-service-ca\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.670507 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.670483 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfb67a-6154-4787-8bae-7c29efde99bc-openshift-service-ca\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.671022 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:35:35.670881 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-serving-cert: secret "maas-api-e2e-ait-a684c9cb-serving-cert" not found Nov 01 01:35:35.671022 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:35:35.670949 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls podName:fadfb67a-6154-4787-8bae-7c29efde99bc nodeName:}" failed. No retries permitted until 2026-11-01 01:35:36.170928967 +0000 UTC m=+4281.046149975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls") pod "maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" (UID: "fadfb67a-6154-4787-8bae-7c29efde99bc") : secret "maas-api-e2e-ait-a684c9cb-serving-cert" not found Nov 01 01:35:35.674618 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.674597 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-metrics-certs\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.732301 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.732257 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsln5\" (UniqueName: \"kubernetes.io/projected/fadfb67a-6154-4787-8bae-7c29efde99bc-kube-api-access-wsln5\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:35.798416 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:35.798378 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:36.010406 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.010375 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:36.175646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.175564 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:36.178140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.178107 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls\") pod \"maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:36.217569 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.217533 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:36.441805 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.441715 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:36.564989 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.564950 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" event={"ID":"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22","Type":"ContainerStarted","Data":"2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6"} Nov 01 01:35:36.565162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.565011 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:36.565162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.565035 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" event={"ID":"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22","Type":"ContainerStarted","Data":"fbee8a367a37c6da599a89c697b52ae0863f769a5419f138cf9784370acb4a7f"} Nov 01 01:35:36.685222 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.685171 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" podStartSLOduration=2.685157084 podStartE2EDuration="2.685157084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:35:36.683308412 +0000 UTC m=+4281.558529419" watchObservedRunningTime="2026-11-01 01:35:36.685157084 +0000 UTC m=+4281.560378099" Nov 01 01:35:36.821392 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:36.821360 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9"] Nov 01 01:35:36.821622 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:35:36.821599 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadfb67a_6154_4787_8bae_7c29efde99bc.slice/crio-5aca3e6421b8f52d56d3331b0fcd6dc2b0385b69ea208901d584b3e95f2683e9 WatchSource:0}: Error finding container 5aca3e6421b8f52d56d3331b0fcd6dc2b0385b69ea208901d584b3e95f2683e9: Status 404 returned error can't find the container with id 5aca3e6421b8f52d56d3331b0fcd6dc2b0385b69ea208901d584b3e95f2683e9 Nov 01 01:35:37.094675 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:37.094581 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w"] Nov 01 01:35:37.094897 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:37.094867 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" podUID="9c4276cd-4570-4993-afb7-a3b23c388b39" containerName="maas-api" containerID="cri-o://34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059" gracePeriod=30 Nov 01 01:35:37.571599 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:37.571551 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" event={"ID":"fadfb67a-6154-4787-8bae-7c29efde99bc","Type":"ContainerStarted","Data":"5aca3e6421b8f52d56d3331b0fcd6dc2b0385b69ea208901d584b3e95f2683e9"} Nov 01 01:35:38.359803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.359772 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 01:35:38.401349 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.401220 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-metrics-certs\") pod \"9c4276cd-4570-4993-afb7-a3b23c388b39\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " Nov 01 01:35:38.401518 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.401365 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4276cd-4570-4993-afb7-a3b23c388b39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4276cd-4570-4993-afb7-a3b23c388b39-openshift-service-ca\") pod \"9c4276cd-4570-4993-afb7-a3b23c388b39\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " Nov 01 01:35:38.401518 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.401405 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-maas-api-tls\") pod \"9c4276cd-4570-4993-afb7-a3b23c388b39\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " Nov 01 01:35:38.401518 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.401435 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4276cd-4570-4993-afb7-a3b23c388b39-kube-api-access-6ppnz\" (UniqueName: \"kubernetes.io/projected/9c4276cd-4570-4993-afb7-a3b23c388b39-kube-api-access-6ppnz\") pod \"9c4276cd-4570-4993-afb7-a3b23c388b39\" (UID: \"9c4276cd-4570-4993-afb7-a3b23c388b39\") " Nov 01 01:35:38.401755 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.401729 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4276cd-4570-4993-afb7-a3b23c388b39-openshift-service-ca" pod "9c4276cd-4570-4993-afb7-a3b23c388b39" (UID: "9c4276cd-4570-4993-afb7-a3b23c388b39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:35:38.401978 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.401958 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4276cd-4570-4993-afb7-a3b23c388b39-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:35:38.403650 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.403627 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-maas-api-tls" pod "9c4276cd-4570-4993-afb7-a3b23c388b39" (UID: "9c4276cd-4570-4993-afb7-a3b23c388b39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:35:38.403749 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.403653 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-metrics-certs" pod "9c4276cd-4570-4993-afb7-a3b23c388b39" (UID: "9c4276cd-4570-4993-afb7-a3b23c388b39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:35:38.403749 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.403697 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4276cd-4570-4993-afb7-a3b23c388b39-kube-api-access-6ppnz" pod "9c4276cd-4570-4993-afb7-a3b23c388b39" (UID: "9c4276cd-4570-4993-afb7-a3b23c388b39"). InnerVolumeSpecName "kube-api-access-6ppnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:35:38.504150 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.504111 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:35:38.504338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.504161 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4276cd-4570-4993-afb7-a3b23c388b39-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:35:38.504338 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.504183 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ppnz\" (UniqueName: \"kubernetes.io/projected/9c4276cd-4570-4993-afb7-a3b23c388b39-kube-api-access-6ppnz\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:35:38.578154 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.578117 6036 generic.go:356] "Generic (PLEG): container finished" podID="9c4276cd-4570-4993-afb7-a3b23c388b39" containerID="34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059" exitCode=0 Nov 01 01:35:38.578378 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.578167 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" Nov 01 01:35:38.578378 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.578211 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" event={"ID":"9c4276cd-4570-4993-afb7-a3b23c388b39","Type":"ContainerDied","Data":"34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059"} Nov 01 01:35:38.578378 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.578262 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w" event={"ID":"9c4276cd-4570-4993-afb7-a3b23c388b39","Type":"ContainerDied","Data":"ee2d4a8dea6131e16564c5446d030e8e36f903ba607e08f19c8fd1bd778296a5"} Nov 01 01:35:38.578378 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.578301 6036 scope.go:122] "RemoveContainer" containerID="34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059" Nov 01 01:35:38.579980 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.579957 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" event={"ID":"fadfb67a-6154-4787-8bae-7c29efde99bc","Type":"ContainerStarted","Data":"affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02"} Nov 01 01:35:38.580274 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.580080 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:38.598165 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.598135 6036 scope.go:122] "RemoveContainer" containerID="34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059" Nov 01 01:35:38.598535 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:35:38.598506 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059\": container with ID starting with 34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059 not found: ID does not exist" containerID="34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059" Nov 01 01:35:38.598654 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.598543 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059"} err="failed to get container status \"34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059\": rpc error: code = NotFound desc = could not find container \"34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059\": container with ID starting with 34dfc5490725a587d74f1a37519c16cd00e1e8c635bdab66b15cd26bbf39e059 not found: ID does not exist" Nov 01 01:35:38.753112 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.753005 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" podStartSLOduration=2.932987365 podStartE2EDuration="3.752992106s" totalImagesPullingTime="820.004741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:35:38.751756216 +0000 UTC m=+4283.626977231" watchObservedRunningTime="2026-11-01 01:35:38.752992106 +0000 UTC m=+4283.628213121" Nov 01 01:35:38.864676 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.864640 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w"] Nov 01 01:35:38.867965 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:38.867939 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c7d8966cd-nxm5w"] Nov 01 01:35:39.249355 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:39.249320 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4276cd-4570-4993-afb7-a3b23c388b39" path="/var/lib/kubelet/pods/9c4276cd-4570-4993-afb7-a3b23c388b39/volumes" Nov 01 01:35:41.024640 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:41.024596 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:44.591955 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:44.591924 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:35:45.166327 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:45.166224 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:45.929457 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:45.929425 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:47.574992 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:47.574958 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:35:49.503969 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:35:49.503931 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:35:50.736580 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:50.736526 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:35:55.845592 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:35:55.845535 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:00.940847 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:00.940810 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:05.824126 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:05.824089 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:10.628479 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:10.628440 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:15.429723 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:15.429678 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:19.365887 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:19.365853 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p"] Nov 01 01:36:19.509427 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:36:19.509387 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:36:19.698504 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:19.698418 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:19.910557 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:19.910517 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:20.233312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:20.233253 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:22.913328 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:22.913257 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:23.175686 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:23.175570 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:24.277032 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.276954 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9"] Nov 01 01:36:24.277490 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.277164 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" podUID="fadfb67a-6154-4787-8bae-7c29efde99bc" containerName="maas-api" containerID="cri-o://affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02" gracePeriod=30 Nov 01 01:36:24.366709 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.366666 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" podUID="9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22" containerName="payload-pre-processing" containerID="cri-o://2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6" gracePeriod=25 Nov 01 01:36:24.434564 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.434528 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:24.629272 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.629246 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:36:24.656458 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.656427 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:36:24.805610 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.805532 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadfb67a-6154-4787-8bae-7c29efde99bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfb67a-6154-4787-8bae-7c29efde99bc-openshift-service-ca\") pod \"fadfb67a-6154-4787-8bae-7c29efde99bc\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " Nov 01 01:36:24.805610 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.805567 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-metrics-certs\") pod \"fadfb67a-6154-4787-8bae-7c29efde99bc\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " Nov 01 01:36:24.805610 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.805588 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-plugins-config-volume\") pod \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\" (UID: \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\") " Nov 01 01:36:24.805610 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.805617 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-kube-api-access-n596q\" (UniqueName: \"kubernetes.io/projected/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-kube-api-access-n596q\") pod \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\" (UID: \"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22\") " Nov 01 01:36:24.805927 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.805737 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadfb67a-6154-4787-8bae-7c29efde99bc-kube-api-access-wsln5\" (UniqueName: \"kubernetes.io/projected/fadfb67a-6154-4787-8bae-7c29efde99bc-kube-api-access-wsln5\") pod \"fadfb67a-6154-4787-8bae-7c29efde99bc\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " Nov 01 01:36:24.805927 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.805792 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls\") pod \"fadfb67a-6154-4787-8bae-7c29efde99bc\" (UID: \"fadfb67a-6154-4787-8bae-7c29efde99bc\") " Nov 01 01:36:24.806048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.805940 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadfb67a-6154-4787-8bae-7c29efde99bc-openshift-service-ca" pod "fadfb67a-6154-4787-8bae-7c29efde99bc" (UID: "fadfb67a-6154-4787-8bae-7c29efde99bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:36:24.806048 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.805979 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-plugins-config-volume" pod "9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22" (UID: "9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:36:24.806580 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.806552 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfb67a-6154-4787-8bae-7c29efde99bc-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:24.806719 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.806583 6036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-plugins-config-volume\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:24.808012 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.807986 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-kube-api-access-n596q" pod "9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22" (UID: "9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22"). InnerVolumeSpecName "kube-api-access-n596q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:36:24.808123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.808086 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-metrics-certs" pod "fadfb67a-6154-4787-8bae-7c29efde99bc" (UID: "fadfb67a-6154-4787-8bae-7c29efde99bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:36:24.808215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.808130 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadfb67a-6154-4787-8bae-7c29efde99bc-kube-api-access-wsln5" pod "fadfb67a-6154-4787-8bae-7c29efde99bc" (UID: "fadfb67a-6154-4787-8bae-7c29efde99bc"). InnerVolumeSpecName "kube-api-access-wsln5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:36:24.808215 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.808168 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls" pod "fadfb67a-6154-4787-8bae-7c29efde99bc" (UID: "fadfb67a-6154-4787-8bae-7c29efde99bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:36:24.822680 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.822650 6036 generic.go:356] "Generic (PLEG): container finished" podID="fadfb67a-6154-4787-8bae-7c29efde99bc" containerID="affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02" exitCode=0 Nov 01 01:36:24.822815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.822701 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" Nov 01 01:36:24.822815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.822726 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" event={"ID":"fadfb67a-6154-4787-8bae-7c29efde99bc","Type":"ContainerDied","Data":"affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02"} Nov 01 01:36:24.822815 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.822806 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" event={"ID":"fadfb67a-6154-4787-8bae-7c29efde99bc","Type":"ContainerDied","Data":"5aca3e6421b8f52d56d3331b0fcd6dc2b0385b69ea208901d584b3e95f2683e9"} Nov 01 01:36:24.822972 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.822834 6036 scope.go:122] "RemoveContainer" containerID="affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02" Nov 01 01:36:24.824231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.824208 6036 generic.go:356] "Generic (PLEG): container finished" podID="9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22" containerID="2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6" exitCode=0 Nov 01 01:36:24.824388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.824248 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" Nov 01 01:36:24.824388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.824246 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" event={"ID":"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22","Type":"ContainerDied","Data":"2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6"} Nov 01 01:36:24.824388 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.824349 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p" event={"ID":"9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22","Type":"ContainerDied","Data":"fbee8a367a37c6da599a89c697b52ae0863f769a5419f138cf9784370acb4a7f"} Nov 01 01:36:24.833637 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.833617 6036 scope.go:122] "RemoveContainer" containerID="affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02" Nov 01 01:36:24.833963 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:36:24.833926 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02\": container with ID starting with affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02 not found: ID does not exist" containerID="affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02" Nov 01 01:36:24.834066 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.833967 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02"} err="failed to get container status \"affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02\": rpc error: code = NotFound desc = could not find container \"affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02\": container with ID starting with affee3873dd43afb9300930add73ab75f141a2accaaabc0f36f26d94c28dfa02 not found: ID does not exist" Nov 01 01:36:24.834066 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.833993 6036 scope.go:122] "RemoveContainer" containerID="2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6" Nov 01 01:36:24.842300 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.842263 6036 scope.go:122] "RemoveContainer" containerID="2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6" Nov 01 01:36:24.842594 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:36:24.842575 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6\": container with ID starting with 2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6 not found: ID does not exist" containerID="2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6" Nov 01 01:36:24.842665 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.842600 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6"} err="failed to get container status \"2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6\": rpc error: code = NotFound desc = could not find container \"2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6\": container with ID starting with 2811196460a8dd90b8027ff4324325d7a60d28abb5077e9e5f105ae67238f5c6 not found: ID does not exist" Nov 01 01:36:24.907816 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.907780 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:24.907816 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.907810 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n596q\" (UniqueName: \"kubernetes.io/projected/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22-kube-api-access-n596q\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:24.907816 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.907822 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsln5\" (UniqueName: \"kubernetes.io/projected/fadfb67a-6154-4787-8bae-7c29efde99bc-kube-api-access-wsln5\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:24.908061 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.907830 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfb67a-6154-4787-8bae-7c29efde99bc-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:24.986446 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.986414 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9"] Nov 01 01:36:24.990614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:24.990586 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9"] Nov 01 01:36:25.101432 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:25.101357 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p"] Nov 01 01:36:25.106040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:25.106015 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a684c9cb-68778b7895-qkf2p"] Nov 01 01:36:25.249760 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:25.249724 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22" path="/var/lib/kubelet/pods/9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22/volumes" Nov 01 01:36:25.250247 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:25.250225 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadfb67a-6154-4787-8bae-7c29efde99bc" path="/var/lib/kubelet/pods/fadfb67a-6154-4787-8bae-7c29efde99bc/volumes" Nov 01 01:36:25.993786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:25.993748 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:26.277352 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:26.277245 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:26.506614 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:26.506581 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:27.586973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:27.586936 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a684c9cb-5fb76b5c9f-5vfw9" podUID="fadfb67a-6154-4787-8bae-7c29efde99bc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.154:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 01:36:29.037097 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:29.037066 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:30.219183 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.219136 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:30.711620 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.711587 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt"] Nov 01 01:36:30.712066 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712053 6036 cpu_manager.go:414] "containerMap: removing container" podUID="fadfb67a-6154-4787-8bae-7c29efde99bc" containerName="maas-api" Nov 01 01:36:30.712162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712067 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadfb67a-6154-4787-8bae-7c29efde99bc" containerName="maas-api" Nov 01 01:36:30.712162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712084 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22" containerName="payload-pre-processing" Nov 01 01:36:30.712162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712089 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22" containerName="payload-pre-processing" Nov 01 01:36:30.712162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712116 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9c4276cd-4570-4993-afb7-a3b23c388b39" containerName="maas-api" Nov 01 01:36:30.712162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712121 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4276cd-4570-4993-afb7-a3b23c388b39" containerName="maas-api" Nov 01 01:36:30.712594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712204 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadfb67a-6154-4787-8bae-7c29efde99bc" containerName="maas-api" Nov 01 01:36:30.712594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712214 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca7b2e2-e635-4fa4-ab3c-ead52ce74d22" containerName="payload-pre-processing" Nov 01 01:36:30.712594 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.712222 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4276cd-4570-4993-afb7-a3b23c388b39" containerName="maas-api" Nov 01 01:36:30.717022 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.717002 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.766790 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.766748 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-token\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.766955 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.766826 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-credential-socket\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.766955 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.766911 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-socket\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.767086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.766956 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-certs\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.767086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.767053 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-envoy\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.767086 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.767072 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istiod-ca-cert\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.767229 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.767104 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-data\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.767339 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.767278 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hm9f\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-kube-api-access-7hm9f\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.767409 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.767386 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-podinfo\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.771009 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.770973 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-930d1a-openshift-default-dockercfg-k4dwr\"" Nov 01 01:36:30.823324 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.823279 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt"] Nov 01 01:36:30.869191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.868771 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-podinfo\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.869191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.868906 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-token\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.869191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.868979 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-credential-socket\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.869191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.869012 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-socket\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.869191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.869060 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-certs\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.869191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.869116 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-envoy\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.869191 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.869143 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istiod-ca-cert\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.869786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.869208 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-data\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.869786 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.869307 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hm9f\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-kube-api-access-7hm9f\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.870607 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.870393 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istiod-ca-cert\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.870607 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.870511 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-data\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.870913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.870664 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-socket\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.870913 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.870792 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-certs\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.871230 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.871193 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-credential-socket\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.873111 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.873088 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-podinfo\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.876155 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.876125 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:30.876539 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.876516 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-envoy\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.933896 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.933857 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-token\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:30.934311 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:30.934263 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hm9f\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-kube-api-access-7hm9f\") pod \"e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:31.030773 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.030691 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:31.128204 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.128164 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:31.462131 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.460971 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt"] Nov 01 01:36:31.464728 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:36:31.464700 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5efc54_cbe1_4e79_bea8_cb02287faa0f.slice/crio-52f43853efc34fa8fc72a6ecb2eb1f2eeb17dd39059577e5eb4e2afce85d299b WatchSource:0}: Error finding container 52f43853efc34fa8fc72a6ecb2eb1f2eeb17dd39059577e5eb4e2afce85d299b: Status 404 returned error can't find the container with id 52f43853efc34fa8fc72a6ecb2eb1f2eeb17dd39059577e5eb4e2afce85d299b Nov 01 01:36:31.467113 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.467073 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:36:31.467204 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.467185 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:36:31.467253 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.467239 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:36:31.865928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.865892 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" event={"ID":"9b5efc54-cbe1-4e79-bea8-cb02287faa0f","Type":"ContainerStarted","Data":"6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b"} Nov 01 01:36:31.866102 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.865974 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" event={"ID":"9b5efc54-cbe1-4e79-bea8-cb02287faa0f","Type":"ContainerStarted","Data":"52f43853efc34fa8fc72a6ecb2eb1f2eeb17dd39059577e5eb4e2afce85d299b"} Nov 01 01:36:31.991167 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:31.991115 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" podStartSLOduration=1.99110145 podStartE2EDuration="1.99110145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:36:31.989399567 +0000 UTC m=+4336.864620583" watchObservedRunningTime="2026-11-01 01:36:31.99110145 +0000 UTC m=+4336.866322465" Nov 01 01:36:32.031914 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:32.031878 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:32.037354 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:32.037328 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:32.870839 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:32.870803 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:32.871767 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:32.871746 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:36:34.870351 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:34.870279 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:35.097049 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:35.097013 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:38.104646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:38.104612 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:38.404708 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:38.404597 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:40.931958 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:40.931924 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz"] Nov 01 01:36:40.935693 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:40.935676 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:40.988970 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:40.988936 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-930d1a-serving-cert\"" Nov 01 01:36:40.994646 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:40.994611 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz"] Nov 01 01:36:41.077896 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.077867 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-metrics-certs\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.078080 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.077924 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl2jc\" (UniqueName: \"kubernetes.io/projected/8a8e4785-4342-4cdb-adb7-4450434474a0-kube-api-access-gl2jc\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.078080 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.078031 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.078213 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.078105 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8e4785-4342-4cdb-adb7-4450434474a0-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.179472 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.179437 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.181670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.179536 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8e4785-4342-4cdb-adb7-4450434474a0-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.181670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.179597 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-metrics-certs\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.181670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.179631 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl2jc\" (UniqueName: \"kubernetes.io/projected/8a8e4785-4342-4cdb-adb7-4450434474a0-kube-api-access-gl2jc\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.181670 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:36:41.179999 6036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-serving-cert: secret "maas-api-e2e-authcfg-http-930d1a-serving-cert" not found Nov 01 01:36:41.181670 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:36:41.180062 6036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls podName:8a8e4785-4342-4cdb-adb7-4450434474a0 nodeName:}" failed. No retries permitted until 2026-11-01 01:36:41.680041456 +0000 UTC m=+4346.555262467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls") pod "maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" (UID: "8a8e4785-4342-4cdb-adb7-4450434474a0") : secret "maas-api-e2e-authcfg-http-930d1a-serving-cert" not found Nov 01 01:36:41.181670 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.180957 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8e4785-4342-4cdb-adb7-4450434474a0-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.183792 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.183696 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-metrics-certs\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.190692 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.190638 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:41.244966 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.244928 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl2jc\" (UniqueName: \"kubernetes.io/projected/8a8e4785-4342-4cdb-adb7-4450434474a0-kube-api-access-gl2jc\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.434232 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.434127 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:41.685533 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.685439 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.688022 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.687990 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:41.849246 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:41.849211 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:42.225140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:42.225114 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz"] Nov 01 01:36:42.227700 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:36:42.227671 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8e4785_4342_4cdb_adb7_4450434474a0.slice/crio-b69ffdbd9973c8ecf9c357665722272a3e0fcee88954ecdce0a51543a69f456f WatchSource:0}: Error finding container b69ffdbd9973c8ecf9c357665722272a3e0fcee88954ecdce0a51543a69f456f: Status 404 returned error can't find the container with id b69ffdbd9973c8ecf9c357665722272a3e0fcee88954ecdce0a51543a69f456f Nov 01 01:36:42.926170 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:42.926132 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" event={"ID":"8a8e4785-4342-4cdb-adb7-4450434474a0","Type":"ContainerStarted","Data":"b69ffdbd9973c8ecf9c357665722272a3e0fcee88954ecdce0a51543a69f456f"} Nov 01 01:36:43.933235 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:43.933194 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" event={"ID":"8a8e4785-4342-4cdb-adb7-4450434474a0","Type":"ContainerStarted","Data":"82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119"} Nov 01 01:36:43.933748 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:43.933268 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:44.076301 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:44.076238 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" podStartSLOduration=3.1812050689999998 podStartE2EDuration="4.076224701s" totalImagesPullingTime="895.019632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:36:44.074358376 +0000 UTC m=+4348.949579413" watchObservedRunningTime="2026-11-01 01:36:44.076224701 +0000 UTC m=+4348.951445716" Nov 01 01:36:45.534419 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:45.534357 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:46.256423 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:46.256389 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:49.504350 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:36:49.504271 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:36:49.945140 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:49.945109 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:36:51.301561 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:51.301530 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:56.156339 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:56.156257 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:36:59.022880 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:59.022849 6036 generic.go:356] "Generic (PLEG): container finished" podID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerID="6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b" exitCode=0 Nov 01 01:36:59.023491 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:59.022927 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" event={"ID":"9b5efc54-cbe1-4e79-bea8-cb02287faa0f","Type":"ContainerDied","Data":"6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b"} Nov 01 01:36:59.023491 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:59.023467 6036 scope.go:122] "RemoveContainer" containerID="6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b" Nov 01 01:36:59.024703 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:59.024676 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:36:59.024773 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:59.024743 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:36:59.024810 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:59.024776 6036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:36:59.915503 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:36:59.915471 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:00.029893 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:00.029862 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" event={"ID":"9b5efc54-cbe1-4e79-bea8-cb02287faa0f","Type":"ContainerStarted","Data":"888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44"} Nov 01 01:37:00.031311 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:00.031294 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:37:00.032325 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:00.032277 6036 patch_prober.go:29] interesting pod/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 01:37:00.032447 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:00.032357 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 01:37:01.031889 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:01.031852 6036 patch_prober.go:29] interesting pod/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 01:37:01.032341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:01.031911 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 01:37:01.363057 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:01.363005 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:02.031248 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:02.031205 6036 patch_prober.go:29] interesting pod/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 01:37:02.031486 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:02.031338 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 01:37:02.870806 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:02.870764 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:37:03.031624 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:03.031590 6036 patch_prober.go:29] interesting pod/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 01:37:03.031810 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:03.031652 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 01:37:04.031444 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:04.031404 6036 patch_prober.go:29] interesting pod/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 01:37:04.031861 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:04.031458 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 01:37:05.031716 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:05.031681 6036 patch_prober.go:29] interesting pod/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 01:37:05.032132 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:05.031735 6036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 01:37:06.036237 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:06.036210 6036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:37:06.037427 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:06.037395 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:37:06.263759 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:06.263704 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:11.139315 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:11.139258 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:15.950429 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:15.950381 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:19.506422 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:37:19.506382 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:37:20.833777 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:20.833744 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:25.629971 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:25.629939 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:30.530250 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:30.530220 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:35.374597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:35.374563 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:40.227642 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:40.227599 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:45.035199 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:45.035166 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:45.862408 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:45.862378 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:49.506191 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:37:49.506150 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:37:49.936943 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:49.936910 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:54.750484 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:54.750400 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:37:59.632837 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:37:59.632806 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:04.442638 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:04.442604 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:09.325826 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:09.325777 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:14.141531 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:14.141498 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:19.061771 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:19.061737 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:19.504867 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:38:19.504782 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:38:24.084951 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:24.084911 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:29.167166 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:29.167129 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:33.216829 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:33.216782 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:33.454105 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:33.454069 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:34.022259 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:34.022220 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:35.564700 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:35.564656 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:35.776264 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:35.776234 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:43.153054 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.153021 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz"] Nov 01 01:38:43.153589 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.153227 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" podUID="8a8e4785-4342-4cdb-adb7-4450434474a0" containerName="maas-api" containerID="cri-o://82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119" gracePeriod=30 Nov 01 01:38:43.315971 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.315931 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:43.422709 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.422682 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:38:43.494125 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.494097 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls\") pod \"8a8e4785-4342-4cdb-adb7-4450434474a0\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " Nov 01 01:38:43.494310 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.494152 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-metrics-certs\") pod \"8a8e4785-4342-4cdb-adb7-4450434474a0\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " Nov 01 01:38:43.494310 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.494271 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8e4785-4342-4cdb-adb7-4450434474a0-kube-api-access-gl2jc\" (UniqueName: \"kubernetes.io/projected/8a8e4785-4342-4cdb-adb7-4450434474a0-kube-api-access-gl2jc\") pod \"8a8e4785-4342-4cdb-adb7-4450434474a0\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " Nov 01 01:38:43.494433 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.494416 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8e4785-4342-4cdb-adb7-4450434474a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8e4785-4342-4cdb-adb7-4450434474a0-openshift-service-ca\") pod \"8a8e4785-4342-4cdb-adb7-4450434474a0\" (UID: \"8a8e4785-4342-4cdb-adb7-4450434474a0\") " Nov 01 01:38:43.494847 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.494781 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8e4785-4342-4cdb-adb7-4450434474a0-openshift-service-ca" pod "8a8e4785-4342-4cdb-adb7-4450434474a0" (UID: "8a8e4785-4342-4cdb-adb7-4450434474a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:38:43.496541 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.496510 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-metrics-certs" pod "8a8e4785-4342-4cdb-adb7-4450434474a0" (UID: "8a8e4785-4342-4cdb-adb7-4450434474a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:38:43.496541 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.496526 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8e4785-4342-4cdb-adb7-4450434474a0-kube-api-access-gl2jc" pod "8a8e4785-4342-4cdb-adb7-4450434474a0" (UID: "8a8e4785-4342-4cdb-adb7-4450434474a0"). InnerVolumeSpecName "kube-api-access-gl2jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:38:43.496705 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.496642 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls" pod "8a8e4785-4342-4cdb-adb7-4450434474a0" (UID: "8a8e4785-4342-4cdb-adb7-4450434474a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:38:43.570108 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.570069 6036 generic.go:356] "Generic (PLEG): container finished" podID="8a8e4785-4342-4cdb-adb7-4450434474a0" containerID="82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119" exitCode=0 Nov 01 01:38:43.570379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.570140 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" Nov 01 01:38:43.570379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.570154 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" event={"ID":"8a8e4785-4342-4cdb-adb7-4450434474a0","Type":"ContainerDied","Data":"82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119"} Nov 01 01:38:43.570379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.570210 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz" event={"ID":"8a8e4785-4342-4cdb-adb7-4450434474a0","Type":"ContainerDied","Data":"b69ffdbd9973c8ecf9c357665722272a3e0fcee88954ecdce0a51543a69f456f"} Nov 01 01:38:43.570379 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.570231 6036 scope.go:122] "RemoveContainer" containerID="82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119" Nov 01 01:38:43.580036 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.580017 6036 scope.go:122] "RemoveContainer" containerID="82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119" Nov 01 01:38:43.580417 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:38:43.580394 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119\": container with ID starting with 82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119 not found: ID does not exist" containerID="82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119" Nov 01 01:38:43.580532 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.580423 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119"} err="failed to get container status \"82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119\": rpc error: code = NotFound desc = could not find container \"82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119\": container with ID starting with 82348f2a096fe0bf26822d4ffeb8d740e4e07aec748a8ee9a0e9057dadf10119 not found: ID does not exist" Nov 01 01:38:43.595809 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.595786 6036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-maas-api-tls\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:43.595809 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.595810 6036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8e4785-4342-4cdb-adb7-4450434474a0-metrics-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:43.595958 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.595822 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl2jc\" (UniqueName: \"kubernetes.io/projected/8a8e4785-4342-4cdb-adb7-4450434474a0-kube-api-access-gl2jc\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:43.595958 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.595832 6036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8e4785-4342-4cdb-adb7-4450434474a0-openshift-service-ca\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:43.737019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.736933 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz"] Nov 01 01:38:43.741513 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:43.741489 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-930d1a-55689dcb86-skbvz"] Nov 01 01:38:44.843424 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:44.843384 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:45.131908 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:45.131828 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:45.249803 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:45.249757 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8e4785-4342-4cdb-adb7-4450434474a0" path="/var/lib/kubelet/pods/8a8e4785-4342-4cdb-adb7-4450434474a0/volumes" Nov 01 01:38:46.000414 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:46.000380 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:46.243944 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:46.243904 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:49.504668 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:38:49.504632 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:38:51.916096 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:51.916068 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:52.163732 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:52.163692 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:52.165472 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:52.165446 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt"] Nov 01 01:38:52.165710 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:52.165687 6036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" containerID="cri-o://888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44" gracePeriod=30 Nov 01 01:38:52.458370 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:52.458316 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:38:57.428089 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.428063 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:38:57.552246 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552214 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-podinfo\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552383 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-envoy\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552431 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-token\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-token\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552480 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552461 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-data\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552657 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552519 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-credential-socket\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552657 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552546 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istiod-ca-cert\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552657 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552619 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-certs\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552817 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552669 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-kube-api-access-7hm9f\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-kube-api-access-7hm9f\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552817 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552702 6036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-socket\") pod \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\" (UID: \"9b5efc54-cbe1-4e79-bea8-cb02287faa0f\") " Nov 01 01:38:57.552930 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552800 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-data" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:38:57.552930 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552835 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-credential-socket" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:38:57.553042 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.552953 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istiod-ca-cert" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:38:57.553103 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.553048 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-certs" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:38:57.553103 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.553083 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-socket" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:38:57.553247 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.553230 6036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-data\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.553337 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.553255 6036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-credential-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.553337 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.553270 6036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istiod-ca-cert\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.553337 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.553302 6036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-certs\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.553337 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.553318 6036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-workload-socket\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.554687 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.554670 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-envoy" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:38:57.555018 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.554994 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-kube-api-access-7hm9f" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "kube-api-access-7hm9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:38:57.555162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.555140 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-token" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:38:57.555162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.555141 6036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-podinfo" pod "9b5efc54-cbe1-4e79-bea8-cb02287faa0f" (UID: "9b5efc54-cbe1-4e79-bea8-cb02287faa0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:38:57.649498 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.649461 6036 generic.go:356] "Generic (PLEG): container finished" podID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerID="888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44" exitCode=0 Nov 01 01:38:57.649693 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.649545 6036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" Nov 01 01:38:57.649693 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.649597 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" event={"ID":"9b5efc54-cbe1-4e79-bea8-cb02287faa0f","Type":"ContainerDied","Data":"888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44"} Nov 01 01:38:57.649693 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.649653 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt" event={"ID":"9b5efc54-cbe1-4e79-bea8-cb02287faa0f","Type":"ContainerDied","Data":"52f43853efc34fa8fc72a6ecb2eb1f2eeb17dd39059577e5eb4e2afce85d299b"} Nov 01 01:38:57.649693 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.649683 6036 scope.go:122] "RemoveContainer" containerID="888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44" Nov 01 01:38:57.654747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.654727 6036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-envoy\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.654747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.654746 6036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-token\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.654938 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.654757 6036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hm9f\" (UniqueName: \"kubernetes.io/projected/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-kube-api-access-7hm9f\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.654938 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.654768 6036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5efc54-cbe1-4e79-bea8-cb02287faa0f-istio-podinfo\") on node \"ip-10-0-141-93.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:57.659414 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.659397 6036 scope.go:122] "RemoveContainer" containerID="6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b" Nov 01 01:38:57.668345 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.668325 6036 scope.go:122] "RemoveContainer" containerID="888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44" Nov 01 01:38:57.668604 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:38:57.668586 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44\": container with ID starting with 888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44 not found: ID does not exist" containerID="888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44" Nov 01 01:38:57.668651 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.668615 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44"} err="failed to get container status \"888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44\": rpc error: code = NotFound desc = could not find container \"888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44\": container with ID starting with 888389b64dc445d92f70c0054a27f885e094a8c7c4bcf0ddc5957dfe7de83b44 not found: ID does not exist" Nov 01 01:38:57.668651 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.668634 6036 scope.go:122] "RemoveContainer" containerID="6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b" Nov 01 01:38:57.668868 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:38:57.668853 6036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b\": container with ID starting with 6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b not found: ID does not exist" containerID="6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b" Nov 01 01:38:57.668905 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.668873 6036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b"} err="failed to get container status \"6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b\": rpc error: code = NotFound desc = could not find container \"6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b\": container with ID starting with 6b984d2499c3da5afa9f5c61490c0d4ec76733e1cbfbad2cd5c5a933b5d4dc3b not found: ID does not exist" Nov 01 01:38:57.816538 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.816455 6036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt"] Nov 01 01:38:57.822050 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:57.822025 6036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-930d1a-openshift-default-755bb59b8f-7bjbt"] Nov 01 01:38:59.249218 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:38:59.249178 6036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" path="/var/lib/kubelet/pods/9b5efc54-cbe1-4e79-bea8-cb02287faa0f/volumes" Nov 01 01:39:19.504036 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:39:19.503994 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:39:44.024701 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:39:44.024621 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:39:44.237306 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:39:44.237258 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:39:46.622297 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:39:46.622249 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:39:49.513019 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:39:49.512973 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:40:06.568467 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:06.568422 6036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8jzfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 01:40:06.569162 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:06.569105 6036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8jzfn" podUID="cb6d7ab3-42e6-40f5-8c71-1d375595043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 01:40:15.246928 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:15.246901 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7784f485d-ftt6z_3f221f59-060d-4828-b652-da453b2558c4/manager/0.log" Nov 01 01:40:16.096203 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:16.096173 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p5lhm_6dd774a7-0574-4eb9-9c7f-a7cff549f713/manager/0.log" Nov 01 01:40:16.900544 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:16.900514 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb6cd4f9b-frmhp_7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7/manager/0.log" Nov 01 01:40:17.334085 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:17.334060 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-z622g_eff1a9a2-2384-4fc2-8487-7f1387461d60/server/0.log" Nov 01 01:40:19.099696 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:19.099668 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b97d8bb58-ll7ch_771d9c89-3e83-4240-8565-2e16cbb4c36c/maas-api/0.log" Nov 01 01:40:19.505045 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:40:19.504963 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:40:22.769143 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:22.769115 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l495l_2826f180-d22f-4ab1-8675-95d4f6b91882/kuadrant-console-plugin/0.log" Nov 01 01:40:25.189601 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:25.189568 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c59cdf59-gbhrr_ec4613c5-e6a3-461f-96ac-a3a2d17e99ae/kube-auth-proxy/0.log" Nov 01 01:40:26.925323 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:26.925294 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-8jzfn_cb6d7ab3-42e6-40f5-8c71-1d375595043e/istio-proxy/0.log" Nov 01 01:40:27.419747 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:27.419709 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-q4kwb_d2e53743-67de-4052-be2e-997144f7a8c1/istio-proxy/0.log" Nov 01 01:40:27.924754 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:27.924725 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-w4d6f_12a82f5c-24ab-4385-9b59-832fec6461a2/istio-proxy/0.log" Nov 01 01:40:30.229088 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:30.229058 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv_44a07f20-d1e3-40cb-aa2d-66ed878c22d5/main/0.log" Nov 01 01:40:30.461954 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:30.461909 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7f5bn_e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04/main/0.log" Nov 01 01:40:31.457973 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:31.457939 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tsnn6_53c3efef-0317-4a09-9edb-3462eac4eb96/main/0.log" Nov 01 01:40:31.939676 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:31.939642 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx_01758a46-b41c-4ac1-b4b7-6baf086384e4/main/0.log" Nov 01 01:40:42.577671 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:42.577631 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7784f485d-ftt6z_3f221f59-060d-4828-b652-da453b2558c4/manager/0.log" Nov 01 01:40:43.238653 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:43.238604 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p5lhm_6dd774a7-0574-4eb9-9c7f-a7cff549f713/manager/0.log" Nov 01 01:40:43.969885 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:43.969856 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb6cd4f9b-frmhp_7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7/manager/0.log" Nov 01 01:40:44.293565 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:44.293537 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-z622g_eff1a9a2-2384-4fc2-8487-7f1387461d60/server/0.log" Nov 01 01:40:45.699384 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:45.699350 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b97d8bb58-ll7ch_771d9c89-3e83-4240-8565-2e16cbb4c36c/maas-api/0.log" Nov 01 01:40:46.939700 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:46.939663 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:40:48.837926 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:48.837896 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l495l_2826f180-d22f-4ab1-8675-95d4f6b91882/kuadrant-console-plugin/0.log" Nov 01 01:40:49.504925 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:40:49.504880 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:40:50.916040 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:50.916009 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c59cdf59-gbhrr_ec4613c5-e6a3-461f-96ac-a3a2d17e99ae/kube-auth-proxy/0.log" Nov 01 01:40:52.600269 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:52.600239 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-8jzfn_cb6d7ab3-42e6-40f5-8c71-1d375595043e/istio-proxy/0.log" Nov 01 01:40:53.099304 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:53.099267 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-q4kwb_d2e53743-67de-4052-be2e-997144f7a8c1/istio-proxy/0.log" Nov 01 01:40:53.592838 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:53.592810 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-w4d6f_12a82f5c-24ab-4385-9b59-832fec6461a2/istio-proxy/0.log" Nov 01 01:40:55.279842 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:55.279817 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l5kvv_44a07f20-d1e3-40cb-aa2d-66ed878c22d5/main/0.log" Nov 01 01:40:55.525123 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:55.525089 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7f5bn_e9f6ff66-a7bf-4aef-9cb1-8b2b86eddb04/main/0.log" Nov 01 01:40:56.499189 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:56.499164 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tsnn6_53c3efef-0317-4a09-9edb-3462eac4eb96/main/0.log" Nov 01 01:40:56.968606 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:40:56.968577 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fgprx_01758a46-b41c-4ac1-b4b7-6baf086384e4/main/0.log" Nov 01 01:41:03.303462 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:41:03.303430 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:41:19.505456 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:41:19.505414 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:41:25.362496 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:41:25.362459 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb6cd4f9b-frmhp_7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7/manager/0.log" Nov 01 01:41:25.954814 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:41:25.954781 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb6cd4f9b-frmhp_7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7/manager/0.log" Nov 01 01:41:47.272612 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:41:47.272580 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:41:49.504471 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:41:49.504437 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:41:58.380221 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:41:58.380077 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:42:04.328099 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:04.328067 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb6cd4f9b-frmhp_7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7/manager/0.log" Nov 01 01:42:04.903252 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:04.903217 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb6cd4f9b-frmhp_7a0e68c4-64bc-4c2b-b35f-1768b2fbf9e7/manager/0.log" Nov 01 01:42:06.355618 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:06.355573 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:42:06.567738 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:06.567707 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:42:16.550363 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:16.550335 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x4r6x_5f953789-9f33-475b-9687-714c6960973c/global-pull-secret-syncer/0.log" Nov 01 01:42:18.498963 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:18.498935 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" Nov 01 01:42:19.504035 ip-10-0-141-93 kubenswrapper[6036]: E1101 01:42:19.503995 6036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9rxd6" containerName="extractor" Nov 01 01:42:20.738312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:20.738251 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-93.us-west-1.compute.internal_57ff4fc49ea0c4ad7e09012148b2302a/haproxy/0.log" Nov 01 01:42:25.401586 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.401500 6036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5"] Nov 01 01:42:25.401982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.401942 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" Nov 01 01:42:25.401982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.401952 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" Nov 01 01:42:25.401982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.401965 6036 cpu_manager.go:414] "containerMap: removing container" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" Nov 01 01:42:25.401982 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.401970 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" Nov 01 01:42:25.402124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.401993 6036 cpu_manager.go:414] "containerMap: removing container" podUID="8a8e4785-4342-4cdb-adb7-4450434474a0" containerName="maas-api" Nov 01 01:42:25.402124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.401998 6036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8e4785-4342-4cdb-adb7-4450434474a0" containerName="maas-api" Nov 01 01:42:25.402124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.402072 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" Nov 01 01:42:25.402124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.402080 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5efc54-cbe1-4e79-bea8-cb02287faa0f" containerName="istio-proxy" Nov 01 01:42:25.402124 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.402091 6036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8e4785-4342-4cdb-adb7-4450434474a0" containerName="maas-api" Nov 01 01:42:25.405300 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.405260 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.458380 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.458347 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd5lg\"/\"openshift-service-ca.crt\"" Nov 01 01:42:25.476718 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.476677 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-sys\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.476950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.476737 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7nv\" (UniqueName: \"kubernetes.io/projected/ab1340d7-bbe1-4818-8128-789672f2bafc-kube-api-access-sl7nv\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.476950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.476819 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-lib-modules\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.476950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.476855 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-podres\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.476950 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.476898 6036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-proc\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.508946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.508915 6036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rd5lg\"/\"default-dockercfg-6z6n4\"" Nov 01 01:42:25.508946 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.508938 6036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd5lg\"/\"kube-root-ca.crt\"" Nov 01 01:42:25.510856 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.510832 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5"] Nov 01 01:42:25.578082 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578042 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-sys\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.578341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578101 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7nv\" (UniqueName: \"kubernetes.io/projected/ab1340d7-bbe1-4818-8128-789672f2bafc-kube-api-access-sl7nv\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.578341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578123 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-lib-modules\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.578341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578166 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-podres\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.578341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578188 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-sys\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.578341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578205 6036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-proc\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.578341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578275 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-podres\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.578341 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578318 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-lib-modules\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.578688 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.578346 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab1340d7-bbe1-4818-8128-789672f2bafc-proc\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.640737 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.640702 6036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7nv\" (UniqueName: \"kubernetes.io/projected/ab1340d7-bbe1-4818-8128-789672f2bafc-kube-api-access-sl7nv\") pod \"perf-node-gather-daemonset-thwx5\" (UID: \"ab1340d7-bbe1-4818-8128-789672f2bafc\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.717750 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.717663 6036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:25.977684 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:25.977610 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l495l_2826f180-d22f-4ab1-8675-95d4f6b91882/kuadrant-console-plugin/0.log" Nov 01 01:42:26.097838 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:26.097811 6036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5"] Nov 01 01:42:26.100108 ip-10-0-141-93 kubenswrapper[6036]: W1101 01:42:26.100073 6036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podab1340d7_bbe1_4818_8128_789672f2bafc.slice/crio-a855ab33914fd2092bceb2943fd7e52f1346f8e0e09ab20d571909b24f1d5a9c WatchSource:0}: Error finding container a855ab33914fd2092bceb2943fd7e52f1346f8e0e09ab20d571909b24f1d5a9c: Status 404 returned error can't find the container with id a855ab33914fd2092bceb2943fd7e52f1346f8e0e09ab20d571909b24f1d5a9c Nov 01 01:42:26.716145 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:26.716109 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" event={"ID":"ab1340d7-bbe1-4818-8128-789672f2bafc","Type":"ContainerStarted","Data":"7fbf7a32d1899826cfa76f7da1be18f6b04852d4b819e91dec695631f31b6488"} Nov 01 01:42:26.716692 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:26.716168 6036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" event={"ID":"ab1340d7-bbe1-4818-8128-789672f2bafc","Type":"ContainerStarted","Data":"a855ab33914fd2092bceb2943fd7e52f1346f8e0e09ab20d571909b24f1d5a9c"} Nov 01 01:42:26.716692 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:26.716194 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:26.833411 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:26.833359 6036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" podStartSLOduration=1.833345038 podStartE2EDuration="1.833345038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 01:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:42:26.832465347 +0000 UTC m=+4691.707686363" watchObservedRunningTime="2026-11-01 01:42:26.833345038 +0000 UTC m=+4691.708566053" Nov 01 01:42:29.939672 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:29.939647 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c292a777-bdf1-483f-945d-73c1ed2f6794/alertmanager/0.log" Nov 01 01:42:30.359464 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:30.359431 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c292a777-bdf1-483f-945d-73c1ed2f6794/config-reloader/0.log" Nov 01 01:42:30.779813 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:30.779731 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c292a777-bdf1-483f-945d-73c1ed2f6794/kube-rbac-proxy-web/0.log" Nov 01 01:42:31.200060 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:31.200033 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c292a777-bdf1-483f-945d-73c1ed2f6794/kube-rbac-proxy/0.log" Nov 01 01:42:31.621916 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:31.621881 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c292a777-bdf1-483f-945d-73c1ed2f6794/kube-rbac-proxy-metric/0.log" Nov 01 01:42:32.042312 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:32.042252 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c292a777-bdf1-483f-945d-73c1ed2f6794/prom-label-proxy/0.log" Nov 01 01:42:32.463267 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:32.463187 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c292a777-bdf1-483f-945d-73c1ed2f6794/init-config-reloader/0.log" Nov 01 01:42:32.731455 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:32.731372 6036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-thwx5" Nov 01 01:42:33.395165 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:33.395137 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qc5gf_dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/kube-state-metrics/0.log" Nov 01 01:42:33.810827 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:33.810797 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qc5gf_dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/kube-rbac-proxy-main/0.log" Nov 01 01:42:34.231503 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:34.231427 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qc5gf_dc5b2a72-a248-4a35-b6f3-e13c5f7b015b/kube-rbac-proxy-self/0.log" Nov 01 01:42:34.656231 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:34.656200 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5db84b6cc6-rp9xt_fbf066bd-0ff4-471d-ba92-d8cd3d6820fe/metrics-server/0.log" Nov 01 01:42:35.078565 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:35.078537 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-d85jb_34995d02-7971-4439-9d1c-c1083cc72be9/monitoring-plugin/0.log" Nov 01 01:42:35.507871 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:35.507843 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5d695_afca2ec8-21f4-4bb6-bc31-009191b47dca/node-exporter/0.log" Nov 01 01:42:35.926893 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:35.926862 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5d695_afca2ec8-21f4-4bb6-bc31-009191b47dca/kube-rbac-proxy/0.log" Nov 01 01:42:36.347019 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:36.346985 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5d695_afca2ec8-21f4-4bb6-bc31-009191b47dca/init-textfile/0.log" Nov 01 01:42:39.322510 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:39.322477 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5grgn_6df8f53e-1025-4a88-8945-4055732800c6/kube-rbac-proxy-main/0.log" Nov 01 01:42:39.742070 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:39.741978 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5grgn_6df8f53e-1025-4a88-8945-4055732800c6/kube-rbac-proxy-self/0.log" Nov 01 01:42:40.161981 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:40.161954 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5grgn_6df8f53e-1025-4a88-8945-4055732800c6/openshift-state-metrics/0.log" Nov 01 01:42:40.592623 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:40.592592 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_866170d7-d2d3-4cff-9266-d9ff163fc2d9/prometheus/0.log" Nov 01 01:42:41.012241 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:41.012157 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_866170d7-d2d3-4cff-9266-d9ff163fc2d9/config-reloader/0.log" Nov 01 01:42:41.431597 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:41.431569 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_866170d7-d2d3-4cff-9266-d9ff163fc2d9/thanos-sidecar/0.log" Nov 01 01:42:41.851205 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:41.851179 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_866170d7-d2d3-4cff-9266-d9ff163fc2d9/kube-rbac-proxy-web/0.log" Nov 01 01:42:42.275437 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:42.275362 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_866170d7-d2d3-4cff-9266-d9ff163fc2d9/kube-rbac-proxy/0.log" Nov 01 01:42:42.695042 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:42.695016 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_866170d7-d2d3-4cff-9266-d9ff163fc2d9/kube-rbac-proxy-thanos/0.log" Nov 01 01:42:43.117169 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:43.117140 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_866170d7-d2d3-4cff-9266-d9ff163fc2d9/init-config-reloader/0.log" Nov 01 01:42:43.542700 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:43.542674 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-6fxrz_24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/prometheus-operator/0.log" Nov 01 01:42:43.960767 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:43.960698 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-6fxrz_24ea6ad3-b91d-4c5c-b0ac-7974d9dab56b/kube-rbac-proxy/0.log" Nov 01 01:42:46.084493 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:46.084458 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546d87964-scgxb_d9deeffc-4dcb-489c-bbbd-f472272d3c13/thanos-query/0.log" Nov 01 01:42:46.505723 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:46.505646 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546d87964-scgxb_d9deeffc-4dcb-489c-bbbd-f472272d3c13/kube-rbac-proxy-web/0.log" Nov 01 01:42:46.924414 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:46.924382 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546d87964-scgxb_d9deeffc-4dcb-489c-bbbd-f472272d3c13/kube-rbac-proxy/0.log" Nov 01 01:42:47.343168 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:47.343141 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546d87964-scgxb_d9deeffc-4dcb-489c-bbbd-f472272d3c13/prom-label-proxy/0.log" Nov 01 01:42:47.628704 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:47.628619 6036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j99d_7ad3371f-f985-4f80-b26f-d69319f24cac/konnectivity-agent/0.log" limit=1024 Nov 01 01:42:47.768187 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:47.768160 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546d87964-scgxb_d9deeffc-4dcb-489c-bbbd-f472272d3c13/kube-rbac-proxy-rules/0.log" Nov 01 01:42:48.186844 ip-10-0-141-93 kubenswrapper[6036]: I1101 01:42:48.186815 6036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546d87964-scgxb_d9deeffc-4dcb-489c-bbbd-f472272d3c13/kube-rbac-proxy-metrics/0.log"